[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:53 dist-test-slave-6wrf krb5kdc[1649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:53 dist-test-slave-6wrf krb5kdc[1649](info): set up 2 sockets
Jun 20 19:04:53 dist-test-slave-6wrf krb5kdc[1649](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 19:04:54.482630 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:04:54 dist-test-slave-6wrf krb5kdc[1649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:04:54.617422 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:46453
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:46453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:54.734493 1662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:54.734794 1662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:54.734866 1662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:54.738966 1662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:54.739056 1662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:54.739096 1662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:54.739172 1662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:54.739203 1662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:54.744474 1662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:46453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:46453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:54.745615 1662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:54.746716 1662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:54.755697 1667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:54.755765 1668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:54.755951 1670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:54.756108 1662 server_base.cc:1034] running on GCE node
I20260620 19:04:54.756837 1662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:54.758095 1662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:54.759414 1662 hybrid_clock.cc:648] HybridClock initialized: now 1781982294759366 us; error 73 us; skew 500 ppm
Jun 20 19:04:54 dist-test-slave-6wrf krb5kdc[1649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982294, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:54.763028 1662 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:04:54.764025 1662 webserver.cc:460] Webserver started at http://127.1.154.190:39083/ using document root <none> and password file <none>
I20260620 19:04:54.764672 1662 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:54.764734 1662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:54.764961 1662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:54.767148 1662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "b4efc5fbbae24fc88ca46510e884b14b"
format_stamp: "Formatted at 2026-06-20 19:04:54 on dist-test-slave-6wrf"
server_key: "278b0649fa6848fd9e9f60fce785d36c"
server_key_iv: "69ffa406a9b632c1f0f262be2e27eb18"
server_key_version: "encryptionkey@0"
I20260620 19:04:54.767742 1662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "b4efc5fbbae24fc88ca46510e884b14b"
format_stamp: "Formatted at 2026-06-20 19:04:54 on dist-test-slave-6wrf"
server_key: "278b0649fa6848fd9e9f60fce785d36c"
server_key_iv: "69ffa406a9b632c1f0f262be2e27eb18"
server_key_version: "encryptionkey@0"
I20260620 19:04:54.772101 1662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:04:54.775394 1677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:54.776984 1662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:04:54.777149 1662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "b4efc5fbbae24fc88ca46510e884b14b"
format_stamp: "Formatted at 2026-06-20 19:04:54 on dist-test-slave-6wrf"
server_key: "278b0649fa6848fd9e9f60fce785d36c"
server_key_iv: "69ffa406a9b632c1f0f262be2e27eb18"
server_key_version: "encryptionkey@0"
I20260620 19:04:54.777268 1662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:54.798182 1662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:54.801587 1662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:54.801852 1662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:54.809788 1662 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:46453
I20260620 19:04:54.809803 1729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:46453 every 8 connection(s)
I20260620 19:04:54.810806 1662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:54.813810 1730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:54.815598 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 1662
I20260620 19:04:54.815812 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:04:54.817183 1642 external_mini_cluster.cc:1428] Setting key 0da12c63d04262d7b4b54ad6cdaff946
I20260620 19:04:54.819548 1730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b: Bootstrap starting.
I20260620 19:04:54.822062 1730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:54.822841 1730 log.cc:824] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:54.825377 1730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b: No bootstrap required, opened a new log
I20260620 19:04:54.830127 1730 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4efc5fbbae24fc88ca46510e884b14b" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46453 } }
I20260620 19:04:54.830391 1730 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:54.830433 1730 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4efc5fbbae24fc88ca46510e884b14b, State: Initialized, Role: FOLLOWER
I20260620 19:04:54.830900 1730 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4efc5fbbae24fc88ca46510e884b14b" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46453 } }
I20260620 19:04:54.831059 1730 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:54.831146 1730 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:54.831269 1730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:54.832242 1730 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4efc5fbbae24fc88ca46510e884b14b" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46453 } }
I20260620 19:04:54.832592 1730 leader_election.cc:302] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4efc5fbbae24fc88ca46510e884b14b; no voters:
I20260620 19:04:54.832930 1730 leader_election.cc:288] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:54.833211 1735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:54.833685 1735 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [term 1 LEADER]: Becoming Leader. State: Replica: b4efc5fbbae24fc88ca46510e884b14b, State: Running, Role: LEADER
I20260620 19:04:54.834219 1735 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4efc5fbbae24fc88ca46510e884b14b" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46453 } }
I20260620 19:04:54.834298 1730 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:54.836766 1736 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4efc5fbbae24fc88ca46510e884b14b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4efc5fbbae24fc88ca46510e884b14b" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46453 } } }
I20260620 19:04:54.836988 1737 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4efc5fbbae24fc88ca46510e884b14b. Latest consensus state: current_term: 1 leader_uuid: "b4efc5fbbae24fc88ca46510e884b14b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4efc5fbbae24fc88ca46510e884b14b" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46453 } } }
I20260620 19:04:54.837536 1737 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:54.837536 1736 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b [sys.catalog]: This master's current role is: LEADER
Jun 20 19:04:54 dist-test-slave-6wrf krb5kdc[1649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
W20260620 19:04:54.840541 1749 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:04:54.840682 1749 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:04:54.840776 1750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:54.841877 1750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:54.848089 1732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:54.820513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.1.154.190:46453)
0620 19:04:54.821056 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:54.821077 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:54.822247 (+ 1170us) server_negotiation.cc:407] Connection header received
0620 19:04:54.825905 (+ 3658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:54.825927 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:54.826339 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:54.826702 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:54.828929 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:54.830527 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:54.832036 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:54.832705 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:54.832876 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:54.840694 (+ 7818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:54.840704 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:54.840719 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:54.840751 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:54.843524 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:54.844386 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:54.844394 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:54.844402 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:54.844542 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:54.844958 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:54.844961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:54.844968 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:54.845375 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:54.845582 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:54.846213 (+ 631us) server_negotiation.cc:299] Negotiation successful
0620 19:04:54.846504 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":167,"threads_started":1}
I20260620 19:04:54.850286 1750 catalog_manager.cc:1347] Generated new cluster ID: 0968cd8c05064b009cf5514583af5011
I20260620 19:04:54.850406 1750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:54.870111 1750 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:54.871444 1750 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:54.876211 1750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4efc5fbbae24fc88ca46510e884b14b: Generated new TSK 0
I20260620 19:04:54.877007 1750 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:04:54.913210 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46453
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:55.042259 1756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:55.042553 1756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:55.042603 1756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:55.046591 1756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:55.046648 1756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:55.046708 1756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:04:55.052032 1756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46453
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:55.053009 1756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:55.054004 1756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:55.064246 1764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:55.064419 1756 server_base.cc:1034] running on GCE node
W20260620 19:04:55.064239 1762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:55.064239 1761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:55.065083 1756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:55.065824 1756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:55.067112 1756 hybrid_clock.cc:648] HybridClock initialized: now 1781982295066975 us; error 126 us; skew 500 ppm
Jun 20 19:04:55 dist-test-slave-6wrf krb5kdc[1649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982295, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:55.071209 1756 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:04:55.072615 1756 webserver.cc:460] Webserver started at http://127.1.154.129:41089/ using document root <none> and password file <none>
I20260620 19:04:55.073258 1756 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:55.073325 1756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:55.073518 1756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:55.075807 1756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "fbd714e2c9084e1d8c189b5f4c2596a0"
format_stamp: "Formatted at 2026-06-20 19:04:55 on dist-test-slave-6wrf"
server_key: "cc8ed59b550c520f1c2f6edb646e6190"
server_key_iv: "347d538e90a7d41827717ccb4f3bbbdb"
server_key_version: "encryptionkey@0"
I20260620 19:04:55.076438 1756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "fbd714e2c9084e1d8c189b5f4c2596a0"
format_stamp: "Formatted at 2026-06-20 19:04:55 on dist-test-slave-6wrf"
server_key: "cc8ed59b550c520f1c2f6edb646e6190"
server_key_iv: "347d538e90a7d41827717ccb4f3bbbdb"
server_key_version: "encryptionkey@0"
I20260620 19:04:55.080332 1756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:04:55.083199 1771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:55.084529 1756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:04:55.084653 1756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "fbd714e2c9084e1d8c189b5f4c2596a0"
format_stamp: "Formatted at 2026-06-20 19:04:55 on dist-test-slave-6wrf"
server_key: "cc8ed59b550c520f1c2f6edb646e6190"
server_key_iv: "347d538e90a7d41827717ccb4f3bbbdb"
server_key_version: "encryptionkey@0"
I20260620 19:04:55.084744 1756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:55.117880 1756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:55.121490 1756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:55.121740 1756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:55.122325 1756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:55.123111 1756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:55.123158 1756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:55.123199 1756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:55.123214 1756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:55.133201 1756 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:37761
I20260620 19:04:55.133246 1884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:37761 every 8 connection(s)
I20260620 19:04:55.134166 1756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:04:55.141347 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 1756
I20260620 19:04:55.141460 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:04:55.141746 1642 external_mini_cluster.cc:1428] Setting key e6a4ffb17f267825360544f14e444bba
Jun 20 19:04:55 dist-test-slave-6wrf krb5kdc[1649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982295, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:04:55.150280 1732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:55.136178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:45961 (local address 127.1.154.190:46453)
0620 19:04:55.136322 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:55.136325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:55.137808 (+ 1483us) server_negotiation.cc:407] Connection header received
0620 19:04:55.139370 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:55.139374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:55.139432 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:55.139513 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:55.140641 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.141279 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.143079 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.143342 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.143471 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:55.146306 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:55.146310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:55.146312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:55.146343 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:55.148165 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:55.148683 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:55.148692 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:55.148693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:55.148768 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:55.149109 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:55.149112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:55.149114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:55.149288 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:55.149402 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:55.149980 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 19:04:55.150106 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:04:55.151290 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:55.136577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:46453 (local address 127.1.154.129:45961)
0620 19:04:55.137375 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:55.137448 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:55.139134 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:55.139678 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:55.139684 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:55.140029 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:55.140488 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:55.140500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.141502 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.141509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:55.142926 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:55.142937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.143509 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.143513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:55.143700 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:55.144346 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:55.144362 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:55.146134 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:55.148289 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:55.148296 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:55.148310 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:55.148571 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:55.148869 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:55.148872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:55.148874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:55.149004 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:55.149399 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:55.149406 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:55.149733 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 19:04:55.149928 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 19:04:55.150245 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
I20260620 19:04:55.152647 1885 heartbeater.cc:347] Connected to a master server at 127.1.154.190:46453
I20260620 19:04:55.153128 1885 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:55.153692 1885 heartbeater.cc:511] Master 127.1.154.190:46453 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
I20260620 19:04:55.155515 1694 ts_manager.cc:194] Registered new tserver with Master: fbd714e2c9084e1d8c189b5f4c2596a0 (127.1.154.129:37761)
I20260620 19:04:55.156687 1694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:45961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:04:55.174783 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46453
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:55.294207 1890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:55.294466 1890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:55.294518 1890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:55.298651 1890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:55.298719 1890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:55.298779 1890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:04:55.304159 1890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46453
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:55.305259 1890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:55.306252 1890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:55.316011 1898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:55.316027 1895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:55.316022 1896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:55.316339 1890 server_base.cc:1034] running on GCE node
I20260620 19:04:55.316962 1890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:55.318252 1890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:55.319489 1890 hybrid_clock.cc:648] HybridClock initialized: now 1781982295319481 us; error 45 us; skew 500 ppm
Jun 20 19:04:55 dist-test-slave-6wrf krb5kdc[1649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982295, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:55.322562 1890 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:04:55.323478 1890 webserver.cc:460] Webserver started at http://127.1.154.130:45019/ using document root <none> and password file <none>
I20260620 19:04:55.324046 1890 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:55.324136 1890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:55.324366 1890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:55.326499 1890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "2c9bb2c0fb1e4c8dbf1930fcbb79f720"
format_stamp: "Formatted at 2026-06-20 19:04:55 on dist-test-slave-6wrf"
server_key: "4624d3ccd41a463074f6055372362c35"
server_key_iv: "c65f0923d648e9de9d2d1b6546bef6e6"
server_key_version: "encryptionkey@0"
I20260620 19:04:55.327044 1890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "2c9bb2c0fb1e4c8dbf1930fcbb79f720"
format_stamp: "Formatted at 2026-06-20 19:04:55 on dist-test-slave-6wrf"
server_key: "4624d3ccd41a463074f6055372362c35"
server_key_iv: "c65f0923d648e9de9d2d1b6546bef6e6"
server_key_version: "encryptionkey@0"
I20260620 19:04:55.331287 1890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 19:04:55.334883 1905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:55.336694 1890 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:04:55.336859 1890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "2c9bb2c0fb1e4c8dbf1930fcbb79f720"
format_stamp: "Formatted at 2026-06-20 19:04:55 on dist-test-slave-6wrf"
server_key: "4624d3ccd41a463074f6055372362c35"
server_key_iv: "c65f0923d648e9de9d2d1b6546bef6e6"
server_key_version: "encryptionkey@0"
I20260620 19:04:55.337013 1890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:55.349447 1890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:55.352979 1890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:55.353274 1890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:55.354162 1890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:55.355487 1890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:55.355614 1890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:55.355692 1890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:55.355739 1890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:55.366096 1890 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:40611
I20260620 19:04:55.366123 2018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:40611 every 8 connection(s)
I20260620 19:04:55.367010 1890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:04:55.372515 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 1890
I20260620 19:04:55.372660 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:04:55.373014 1642 external_mini_cluster.cc:1428] Setting key 6c0ef9e6fe306c1a5edc2f79581c061f
Jun 20 19:04:55 dist-test-slave-6wrf krb5kdc[1649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982295, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
I20260620 19:04:55.384404 1732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:55.369543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:55271 (local address 127.1.154.190:46453)
0620 19:04:55.369845 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:55.369850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:55.370955 (+ 1105us) server_negotiation.cc:407] Connection header received
0620 19:04:55.372849 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:55.372852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:55.372910 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:55.373011 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:55.374768 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.375289 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.377307 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.377732 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.377850 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:55.380597 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:55.380603 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:55.380609 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:55.380641 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:55.382270 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:55.382835 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:55.382840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:55.382842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:55.382923 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:55.383267 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:55.383272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:55.383274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:55.383457 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:55.383573 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:55.384116 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 19:04:55.384238 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260620 19:04:55.385430 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:55.369958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:46453 (local address 127.1.154.130:55271)
0620 19:04:55.370724 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:55.370784 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:55.372526 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:55.373299 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:55.373310 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:55.373885 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:55.374589 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:55.374605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.375431 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.375438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:55.377141 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:55.377159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.377920 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.377923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:55.378080 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:55.378764 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:55.378781 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:55.380387 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:55.382394 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:55.382402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:55.382419 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:55.382725 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:55.383018 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:55.383022 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:55.383024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:55.383184 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:55.383571 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:55.383579 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:55.383865 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 19:04:55.384070 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 19:04:55.384457 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
I20260620 19:04:55.386763 2019 heartbeater.cc:347] Connected to a master server at 127.1.154.190:46453
I20260620 19:04:55.387001 2019 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:55.387506 2019 heartbeater.cc:511] Master 127.1.154.190:46453 requested a full tablet report, sending...
I20260620 19:04:55.388576 1694 ts_manager.cc:194] Registered new tserver with Master: 2c9bb2c0fb1e4c8dbf1930fcbb79f720 (127.1.154.130:40611)
I20260620 19:04:55.389376 1694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:55271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:04:55.405737 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46453
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:55.528779 2024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:55.529064 2024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:55.529110 2024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:55.533294 2024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:55.533360 2024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:55.533421 2024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:04:55.538981 2024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46453
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:55.540112 2024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:55.541206 2024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:55.551055 2029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:55.551077 2032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:55.551056 2030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:55.551285 2024 server_base.cc:1034] running on GCE node
I20260620 19:04:55.551857 2024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:55.552500 2024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:55.553712 2024 hybrid_clock.cc:648] HybridClock initialized: now 1781982295553678 us; error 46 us; skew 500 ppm
Jun 20 19:04:55 dist-test-slave-6wrf krb5kdc[1649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982295, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:55.556937 2024 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:04:55.557927 2024 webserver.cc:460] Webserver started at http://127.1.154.131:35611/ using document root <none> and password file <none>
I20260620 19:04:55.558480 2024 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:55.558542 2024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:55.558763 2024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:55.560788 2024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "3698920e37364d9f86b23d83420e70ad"
format_stamp: "Formatted at 2026-06-20 19:04:55 on dist-test-slave-6wrf"
server_key: "21fc680b173f95d443d9a89776b6d5af"
server_key_iv: "be49d04cd6723bd93863a6e890d0dba0"
server_key_version: "encryptionkey@0"
I20260620 19:04:55.561370 2024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "3698920e37364d9f86b23d83420e70ad"
format_stamp: "Formatted at 2026-06-20 19:04:55 on dist-test-slave-6wrf"
server_key: "21fc680b173f95d443d9a89776b6d5af"
server_key_iv: "be49d04cd6723bd93863a6e890d0dba0"
server_key_version: "encryptionkey@0"
I20260620 19:04:55.565330 2024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 19:04:55.568445 2039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:55.570271 2024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:04:55.570387 2024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "3698920e37364d9f86b23d83420e70ad"
format_stamp: "Formatted at 2026-06-20 19:04:55 on dist-test-slave-6wrf"
server_key: "21fc680b173f95d443d9a89776b6d5af"
server_key_iv: "be49d04cd6723bd93863a6e890d0dba0"
server_key_version: "encryptionkey@0"
I20260620 19:04:55.570538 2024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:55.584959 2024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:55.588588 2024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:55.588755 2024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:55.589315 2024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:55.590265 2024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:55.590345 2024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:55.590421 2024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:55.590476 2024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:55.600528 2024 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:46055
I20260620 19:04:55.600585 2152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:46055 every 8 connection(s)
I20260620 19:04:55.601557 2024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:04:55.603910 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 2024
I20260620 19:04:55.604038 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:04:55.604377 1642 external_mini_cluster.cc:1428] Setting key 0bd642213d15bffe69f382bd5c9cff85
Jun 20 19:04:55 dist-test-slave-6wrf krb5kdc[1649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982295, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:04:55.619022 1732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:55.603507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:38693 (local address 127.1.154.190:46453)
0620 19:04:55.603695 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:55.603700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:55.604612 (+ 912us) server_negotiation.cc:407] Connection header received
0620 19:04:55.606162 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:55.606165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:55.606218 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:55.606296 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:55.607542 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.608154 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.609417 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.609767 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.609868 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:55.613706 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:55.613711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:55.613714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:55.613759 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:55.615572 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:55.616360 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:55.616364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:55.616365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:55.616576 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:55.617252 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:55.617258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:55.617261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:55.617578 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:55.617740 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:55.618629 (+ 889us) server_negotiation.cc:299] Negotiation successful
0620 19:04:55.618826 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:04:55.620327 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:55.603868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:46453 (local address 127.1.154.131:38693)
0620 19:04:55.604436 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:55.604496 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:55.605952 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:55.606489 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:55.606495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:55.606871 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:55.607357 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:55.607369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.608296 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.608298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:55.609267 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:55.609276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.609917 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.609920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:55.610465 (+ 545us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:55.611338 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:55.611358 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:55.613363 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:55.615781 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:55.615788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:55.615807 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:55.616145 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:55.616766 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:55.616770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:55.616772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:55.617067 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:55.617776 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:55.617807 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:55.618292 (+ 485us) client_negotiation.cc:769] Sending connection context
0620 19:04:55.618568 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 19:04:55.619016 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":165,"threads_started":1}
I20260620 19:04:55.621527 2153 heartbeater.cc:347] Connected to a master server at 127.1.154.190:46453
I20260620 19:04:55.621815 2153 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:55.622344 2153 heartbeater.cc:511] Master 127.1.154.190:46453 requested a full tablet report, sending...
I20260620 19:04:55.623652 1694 ts_manager.cc:194] Registered new tserver with Master: 3698920e37364d9f86b23d83420e70ad (127.1.154.131:46055)
I20260620 19:04:55.624218 1694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:38693
I20260620 19:04:55.632543 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:04:55 dist-test-slave-6wrf krb5kdc[1649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:04:55 dist-test-slave-6wrf krb5kdc[1649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.154.129@KRBTEST.COM
I20260620 19:04:55.665796 2160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:55.653552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44290 (local address 127.1.154.129:37761)
0620 19:04:55.654079 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:55.654091 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:55.654264 (+ 173us) server_negotiation.cc:407] Connection header received
0620 19:04:55.654375 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:55.654383 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:55.654634 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:55.654824 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:55.655602 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.656443 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.657401 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.657913 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.658039 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:55.661227 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:55.661344 (+ 117us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:55.661366 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:55.661406 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:55.663617 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:55.664250 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:55.664257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:55.664258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:55.664348 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:55.664646 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:55.664649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:55.664651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:55.664948 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:55.665170 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:55.665418 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 19:04:55.665629 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":176,"threads_started":1}
W20260620 19:04:55.666937 1799 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44290
Jun 20 19:04:55 dist-test-slave-6wrf krb5kdc[1649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:04:55 dist-test-slave-6wrf krb5kdc[1649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.129@KRBTEST.COM
I20260620 19:04:55.697935 2160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:55.688082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.1.154.129:37761)
0620 19:04:55.688353 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:55.688361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:55.688444 (+ 83us) server_negotiation.cc:407] Connection header received
0620 19:04:55.688586 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:55.688590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:55.688649 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:55.688760 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:55.689581 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.690228 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.691479 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:55.691710 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:55.691797 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:55.694347 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:55.694350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:55.694352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:55.694379 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:55.695985 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:55.696756 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:55.696759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:55.696760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:55.696840 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:55.697149 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:55.697151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:55.697152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:55.697327 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:55.697428 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:55.697702 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 19:04:55.697811 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260620 19:04:55.699577 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 1756
I20260620 19:04:55.705361 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 1890
I20260620 19:04:55.712582 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 2024
I20260620 19:04:55.719266 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 1662
2026-06-20T19:04:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2308 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:55 dist-test-slave-6wrf krb5kdc[2167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:55 dist-test-slave-6wrf krb5kdc[2167](info): set up 2 sockets
Jun 20 19:04:55 dist-test-slave-6wrf krb5kdc[2167](info): commencing operation
krb5kdc: starting...
W20260620 19:04:56.793962 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:04:56 dist-test-slave-6wrf krb5kdc[2167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:04:56.934342 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:40905
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:40905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:57.048071 2180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:57.048452 2180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:57.048568 2180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:57.054261 2180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:57.054347 2180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:57.054375 2180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:57.054416 2180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:57.054445 2180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:57.063817 2180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:40905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:40905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:57.064934 2180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:57.066012 2180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:57.076629 2186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:57.076592 2185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:57.076656 2188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:57.076851 2180 server_base.cc:1034] running on GCE node
I20260620 19:04:57.077415 2180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:57.078614 2180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:57.079846 2180 hybrid_clock.cc:648] HybridClock initialized: now 1781982297079838 us; error 55 us; skew 500 ppm
Jun 20 19:04:57 dist-test-slave-6wrf krb5kdc[2167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982297, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:57.083487 2180 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:04:57.084438 2180 webserver.cc:460] Webserver started at http://127.1.154.190:46855/ using document root <none> and password file <none>
I20260620 19:04:57.085011 2180 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:57.085098 2180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:57.085330 2180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:57.087464 2180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "e02c80001f4f4556a866d54f6642ce65"
format_stamp: "Formatted at 2026-06-20 19:04:57 on dist-test-slave-6wrf"
server_key: "273b63fdd26bc2c9800bdfe91a093ce3"
server_key_iv: "7deb8d9d84007638b4b72dca695ce352"
server_key_version: "encryptionkey@0"
I20260620 19:04:57.088013 2180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "e02c80001f4f4556a866d54f6642ce65"
format_stamp: "Formatted at 2026-06-20 19:04:57 on dist-test-slave-6wrf"
server_key: "273b63fdd26bc2c9800bdfe91a093ce3"
server_key_iv: "7deb8d9d84007638b4b72dca695ce352"
server_key_version: "encryptionkey@0"
I20260620 19:04:57.091895 2180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:04:57.094724 2195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:57.096845 2180 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:04:57.096999 2180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "e02c80001f4f4556a866d54f6642ce65"
format_stamp: "Formatted at 2026-06-20 19:04:57 on dist-test-slave-6wrf"
server_key: "273b63fdd26bc2c9800bdfe91a093ce3"
server_key_iv: "7deb8d9d84007638b4b72dca695ce352"
server_key_version: "encryptionkey@0"
I20260620 19:04:57.097159 2180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:57.112440 2180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:57.117062 2180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:57.117268 2180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:57.125643 2180 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:40905
I20260620 19:04:57.125692 2247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:40905 every 8 connection(s)
I20260620 19:04:57.126675 2180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:57.130785 2248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:57.133427 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 2180
I20260620 19:04:57.133587 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:04:57.134306 1642 external_mini_cluster.cc:1428] Setting key 0d1149d7f841e8e3aa21f5c3302316c9
I20260620 19:04:57.139010 2248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65: Bootstrap starting.
I20260620 19:04:57.141376 2248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:57.142307 2248 log.cc:824] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:04:57 dist-test-slave-6wrf krb5kdc[2167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:04:57.144578 2248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65: No bootstrap required, opened a new log
I20260620 19:04:57.148867 2248 raft_consensus.cc:348] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02c80001f4f4556a866d54f6642ce65" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40905 } }
I20260620 19:04:57.149248 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:57.136096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.1.154.190:40905)
0620 19:04:57.136623 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:57.136639 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:57.136678 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:04:57.137375 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:57.137391 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:57.137834 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:57.138190 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:57.139016 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.140072 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.140934 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.141418 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.141594 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:57.144511 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:57.144516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:57.144528 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:57.144555 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:57.146383 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:57.146847 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:57.146858 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:57.146862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:57.146968 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:57.147253 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:57.147258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:57.147260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:57.147659 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:57.147878 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:57.148150 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 19:04:57.148439 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":160,"threads_started":1}
I20260620 19:04:57.149305 2248 raft_consensus.cc:374] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:57.149703 2248 raft_consensus.cc:729] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02c80001f4f4556a866d54f6642ce65, State: Initialized, Role: FOLLOWER
I20260620 19:04:57.150230 2248 consensus_queue.cc:260] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02c80001f4f4556a866d54f6642ce65" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40905 } }
I20260620 19:04:57.150430 2248 raft_consensus.cc:388] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:57.150524 2248 raft_consensus.cc:482] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:57.150628 2248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:57.151597 2248 raft_consensus.cc:504] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02c80001f4f4556a866d54f6642ce65" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40905 } }
I20260620 19:04:57.151968 2248 leader_election.cc:302] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e02c80001f4f4556a866d54f6642ce65; no voters:
I20260620 19:04:57.152277 2248 leader_election.cc:288] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:57.152456 2253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:57.152704 2253 raft_consensus.cc:686] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [term 1 LEADER]: Becoming Leader. State: Replica: e02c80001f4f4556a866d54f6642ce65, State: Running, Role: LEADER
I20260620 19:04:57.153005 2253 consensus_queue.cc:237] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02c80001f4f4556a866d54f6642ce65" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40905 } }
I20260620 19:04:57.153582 2248 sys_catalog.cc:564] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:57.154804 2255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e02c80001f4f4556a866d54f6642ce65. Latest consensus state: current_term: 1 leader_uuid: "e02c80001f4f4556a866d54f6642ce65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02c80001f4f4556a866d54f6642ce65" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40905 } } }
I20260620 19:04:57.154953 2255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:57.155273 2254 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e02c80001f4f4556a866d54f6642ce65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02c80001f4f4556a866d54f6642ce65" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40905 } } }
I20260620 19:04:57.155362 2254 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:57.155443 2262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:57.158684 2262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:57.165159 2262 catalog_manager.cc:1347] Generated new cluster ID: bce5c105b4d34717b37b579537432d87
I20260620 19:04:57.165342 2262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:57.182833 2262 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:57.184264 2262 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:57.193147 2262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e02c80001f4f4556a866d54f6642ce65: Generated new TSK 0
I20260620 19:04:57.194020 2262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:04:57.229442 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40905
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:57.350584 2274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:57.350915 2274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:57.351002 2274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:57.355368 2274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:57.355468 2274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:57.355556 2274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:04:57.361204 2274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40905
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:57.362263 2274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:57.363272 2274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:57.373647 2280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:57.373713 2282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:57.373716 2279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:57.373706 2274 server_base.cc:1034] running on GCE node
I20260620 19:04:57.374336 2274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:57.374984 2274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:57.376149 2274 hybrid_clock.cc:648] HybridClock initialized: now 1781982297376137 us; error 33 us; skew 500 ppm
Jun 20 19:04:57 dist-test-slave-6wrf krb5kdc[2167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982297, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:57.378978 2274 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:04:57.379906 2274 webserver.cc:460] Webserver started at http://127.1.154.129:44467/ using document root <none> and password file <none>
I20260620 19:04:57.380448 2274 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:57.380507 2274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:57.380666 2274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:57.382788 2274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "135d318c5a84403682c25460a081763a"
format_stamp: "Formatted at 2026-06-20 19:04:57 on dist-test-slave-6wrf"
server_key: "d69d97690cfd07a6454e6fc3948ef4b2"
server_key_iv: "a86cbc411b0dae136329e8ad95f56ee7"
server_key_version: "encryptionkey@0"
I20260620 19:04:57.383302 2274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "135d318c5a84403682c25460a081763a"
format_stamp: "Formatted at 2026-06-20 19:04:57 on dist-test-slave-6wrf"
server_key: "d69d97690cfd07a6454e6fc3948ef4b2"
server_key_iv: "a86cbc411b0dae136329e8ad95f56ee7"
server_key_version: "encryptionkey@0"
I20260620 19:04:57.387135 2274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:04:57.389647 2289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:57.391441 2274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:04:57.391623 2274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "135d318c5a84403682c25460a081763a"
format_stamp: "Formatted at 2026-06-20 19:04:57 on dist-test-slave-6wrf"
server_key: "d69d97690cfd07a6454e6fc3948ef4b2"
server_key_iv: "a86cbc411b0dae136329e8ad95f56ee7"
server_key_version: "encryptionkey@0"
I20260620 19:04:57.391754 2274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:57.405156 2274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:57.409127 2274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:57.409368 2274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:57.410063 2274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:57.410928 2274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:57.411000 2274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:57.411072 2274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:57.411124 2274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:57.421161 2274 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:34953
I20260620 19:04:57.421186 2402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:34953 every 8 connection(s)
I20260620 19:04:57.422102 2274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:04:57.426865 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 2274
I20260620 19:04:57.426997 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:04:57.427282 1642 external_mini_cluster.cc:1428] Setting key fcb7bd4326d72d8c6f6445e9bea4de98
Jun 20 19:04:57 dist-test-slave-6wrf krb5kdc[2167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982297, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
I20260620 19:04:57.439154 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:57.424438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:49859 (local address 127.1.154.190:40905)
0620 19:04:57.424580 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:57.424584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:57.425547 (+ 963us) server_negotiation.cc:407] Connection header received
0620 19:04:57.427200 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:57.427208 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:57.427301 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:57.427465 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:57.429290 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.429900 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.431080 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.431303 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.431421 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:57.434408 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:57.434414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:57.434418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:57.434456 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:57.436719 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:57.437386 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:57.437389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:57.437391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:57.437477 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:57.437939 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:57.437943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:57.437944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:57.438140 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:57.438220 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:57.438901 (+ 681us) server_negotiation.cc:299] Negotiation successful
0620 19:04:57.439022 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:04:57.439986 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:57.424524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:40905 (local address 127.1.154.129:49859)
0620 19:04:57.425292 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:57.425337 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:57.426850 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:57.427796 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:57.427807 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:57.428476 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:57.429115 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:57.429134 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.430047 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.430051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:57.430939 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:57.430947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.431563 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.431573 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:57.431747 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:57.432390 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:57.432407 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:57.434199 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:57.436856 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:57.436864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:57.436882 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:57.437261 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:57.437585 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:57.437590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:57.437612 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:57.437796 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:57.438250 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:57.438257 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:57.438598 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 19:04:57.438813 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 19:04:57.439095 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
I20260620 19:04:57.441551 2403 heartbeater.cc:347] Connected to a master server at 127.1.154.190:40905
I20260620 19:04:57.441849 2403 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:57.442363 2403 heartbeater.cc:511] Master 127.1.154.190:40905 requested a full tablet report, sending...
I20260620 19:04:57.443958 2212 ts_manager.cc:194] Registered new tserver with Master: 135d318c5a84403682c25460a081763a (127.1.154.129:34953)
I20260620 19:04:57.445127 2212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:49859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:04:57.456987 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40905
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:57.570828 2408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:57.571481 2408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:57.571552 2408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:57.575906 2408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:57.575973 2408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:57.576036 2408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:04:57.581439 2408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40905
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:57.582572 2408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:57.583608 2408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:57.594301 2416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:57.594515 2408 server_base.cc:1034] running on GCE node
W20260620 19:04:57.594301 2413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:57.594329 2414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:57.595121 2408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:57.595786 2408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:57.596985 2408 hybrid_clock.cc:648] HybridClock initialized: now 1781982297596957 us; error 55 us; skew 500 ppm
Jun 20 19:04:57 dist-test-slave-6wrf krb5kdc[2167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982297, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:57.600319 2408 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:04:57.601848 2408 webserver.cc:460] Webserver started at http://127.1.154.130:37927/ using document root <none> and password file <none>
I20260620 19:04:57.602473 2408 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:57.602563 2408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:57.602780 2408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:57.604844 2408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "f5533734bb2143c3b5d6b102c32e355f"
format_stamp: "Formatted at 2026-06-20 19:04:57 on dist-test-slave-6wrf"
server_key: "b6dde6e7eab92c953142a182194f3f9d"
server_key_iv: "5ee11715789955e89f25bd946bbfbc5d"
server_key_version: "encryptionkey@0"
I20260620 19:04:57.605410 2408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "f5533734bb2143c3b5d6b102c32e355f"
format_stamp: "Formatted at 2026-06-20 19:04:57 on dist-test-slave-6wrf"
server_key: "b6dde6e7eab92c953142a182194f3f9d"
server_key_iv: "5ee11715789955e89f25bd946bbfbc5d"
server_key_version: "encryptionkey@0"
I20260620 19:04:57.611071 2408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 19:04:57.614516 2423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:57.617579 2408 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260620 19:04:57.617758 2408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "f5533734bb2143c3b5d6b102c32e355f"
format_stamp: "Formatted at 2026-06-20 19:04:57 on dist-test-slave-6wrf"
server_key: "b6dde6e7eab92c953142a182194f3f9d"
server_key_iv: "5ee11715789955e89f25bd946bbfbc5d"
server_key_version: "encryptionkey@0"
I20260620 19:04:57.618417 2408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:57.638969 2408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:57.643401 2408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:57.643610 2408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:57.644367 2408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:57.645522 2408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:57.645587 2408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:57.645682 2408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:57.645715 2408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:57.658818 2408 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:37719
I20260620 19:04:57.658874 2536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:37719 every 8 connection(s)
I20260620 19:04:57.660506 2408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:04:57.664554 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 2408
I20260620 19:04:57.664681 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:04:57.665001 1642 external_mini_cluster.cc:1428] Setting key 9cf7cccdc09306bf1b688ba8336515b7
Jun 20 19:04:57 dist-test-slave-6wrf krb5kdc[2167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982297, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
I20260620 19:04:57.678336 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:57.662828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:33719 (local address 127.1.154.190:40905)
0620 19:04:57.663000 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:57.663006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:57.664120 (+ 1114us) server_negotiation.cc:407] Connection header received
0620 19:04:57.666249 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:57.666254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:57.666317 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:57.666380 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:57.668361 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.669238 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.670713 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.670931 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.671025 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:57.674118 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:57.674122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:57.674125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:57.674152 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:57.676142 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:57.676701 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:57.676704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:57.676706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:57.676786 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:57.677135 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:57.677138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:57.677140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:57.677331 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:57.677452 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:57.678037 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 19:04:57.678159 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 19:04:57.679253 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:57.663326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:40905 (local address 127.1.154.130:33719)
0620 19:04:57.663947 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:57.664010 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:57.665978 (+ 1968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:57.666835 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:57.666847 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:57.667416 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:57.668176 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:57.668198 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.669410 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.669414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:57.670580 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:57.670589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.671116 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.671123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:57.671346 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:57.672083 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:57.672108 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:57.673943 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:57.676298 (+ 2355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:57.676304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:57.676321 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:57.676588 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:57.676894 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:57.676897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:57.676899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:57.677035 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:57.677450 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:57.677458 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:57.677781 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 19:04:57.677984 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 19:04:57.678314 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":147,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
I20260620 19:04:57.680733 2537 heartbeater.cc:347] Connected to a master server at 127.1.154.190:40905
I20260620 19:04:57.681018 2537 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:57.681648 2537 heartbeater.cc:511] Master 127.1.154.190:40905 requested a full tablet report, sending...
I20260620 19:04:57.683071 2212 ts_manager.cc:194] Registered new tserver with Master: f5533734bb2143c3b5d6b102c32e355f (127.1.154.130:37719)
I20260620 19:04:57.684006 2212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:33719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:04:57.697242 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40905
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:57.809185 2542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:57.809448 2542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:57.809489 2542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:57.813457 2542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:57.813524 2542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:57.813582 2542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:04:57.818867 2542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40905
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:57.819921 2542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:57.820915 2542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:57.830662 2547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:57.830613 2550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:57.830621 2548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:57.831072 2542 server_base.cc:1034] running on GCE node
I20260620 19:04:57.831440 2542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:57.832029 2542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:57.833302 2542 hybrid_clock.cc:648] HybridClock initialized: now 1781982297833265 us; error 50 us; skew 500 ppm
Jun 20 19:04:57 dist-test-slave-6wrf krb5kdc[2167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982297, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:57.836359 2542 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:04:57.837284 2542 webserver.cc:460] Webserver started at http://127.1.154.131:33957/ using document root <none> and password file <none>
I20260620 19:04:57.837937 2542 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:57.838027 2542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:57.838255 2542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:57.840255 2542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "466c0286d9fc439399556a697f77b9ce"
format_stamp: "Formatted at 2026-06-20 19:04:57 on dist-test-slave-6wrf"
server_key: "6aff8bea08895b0821cae425f3a27860"
server_key_iv: "a702c06de67b001370b993b0a92bd43c"
server_key_version: "encryptionkey@0"
I20260620 19:04:57.840837 2542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "466c0286d9fc439399556a697f77b9ce"
format_stamp: "Formatted at 2026-06-20 19:04:57 on dist-test-slave-6wrf"
server_key: "6aff8bea08895b0821cae425f3a27860"
server_key_iv: "a702c06de67b001370b993b0a92bd43c"
server_key_version: "encryptionkey@0"
I20260620 19:04:57.844631 2542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:04:57.847206 2557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:57.848510 2542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:04:57.848663 2542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "466c0286d9fc439399556a697f77b9ce"
format_stamp: "Formatted at 2026-06-20 19:04:57 on dist-test-slave-6wrf"
server_key: "6aff8bea08895b0821cae425f3a27860"
server_key_iv: "a702c06de67b001370b993b0a92bd43c"
server_key_version: "encryptionkey@0"
I20260620 19:04:57.848778 2542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:57.866286 2542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:57.870021 2542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:57.870230 2542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:57.870854 2542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:57.871699 2542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:57.871779 2542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:57.871850 2542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:57.871904 2542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:57.881395 2542 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:44847
I20260620 19:04:57.881412 2670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:44847 every 8 connection(s)
I20260620 19:04:57.882323 2542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:04:57.884296 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 2542
I20260620 19:04:57.884438 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:04:57.884719 1642 external_mini_cluster.cc:1428] Setting key 40d5a1c022a371220be0ce0fd988524a
Jun 20 19:04:57 dist-test-slave-6wrf krb5kdc[2167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982297, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:04:57.896879 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:57.884185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:41945 (local address 127.1.154.190:40905)
0620 19:04:57.884352 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:57.884357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:57.885759 (+ 1402us) server_negotiation.cc:407] Connection header received
0620 19:04:57.887304 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:57.887307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:57.887357 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:57.887433 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:57.888552 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.889109 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.890333 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.890583 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.890701 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:57.893193 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:57.893196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:57.893198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:57.893219 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:57.894774 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:57.895337 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:57.895341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:57.895343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:57.895424 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:57.895775 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:57.895780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:57.895782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:57.896012 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:57.896119 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:57.896593 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 19:04:57.896703 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 19:04:57.897939 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:57.884662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:40905 (local address 127.1.154.131:41945)
0620 19:04:57.885582 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:57.885669 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:57.887113 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:57.887592 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:57.887598 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:57.887944 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:57.888398 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:57.888426 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.889250 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.889253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:57.890205 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:57.890215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.890734 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.890738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:57.890915 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:57.891515 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:57.891531 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:57.893015 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:57.894933 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:57.894939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:57.894953 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:57.895215 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:57.895524 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:57.895528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:57.895530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:57.895680 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:57.896117 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:57.896123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:57.896364 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 19:04:57.896554 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 19:04:57.896865 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":159,"threads_started":1}
I20260620 19:04:57.899014 2671 heartbeater.cc:347] Connected to a master server at 127.1.154.190:40905
I20260620 19:04:57.899251 2671 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:57.899737 2671 heartbeater.cc:511] Master 127.1.154.190:40905 requested a full tablet report, sending...
I20260620 19:04:57.900681 2212 ts_manager.cc:194] Registered new tserver with Master: 466c0286d9fc439399556a697f77b9ce (127.1.154.131:44847)
I20260620 19:04:57.901438 2212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:41945
I20260620 19:04:57.911677 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:04:57.925917 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:57.916518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60844 (local address 127.1.154.190:40905)
0620 19:04:57.916690 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:57.916694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:57.916952 (+ 258us) server_negotiation.cc:407] Connection header received
0620 19:04:57.917032 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:57.917035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:57.917189 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:57.917317 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:57.918184 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.918745 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.919931 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.920370 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.920481 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:57.922099 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:57.922102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:57.922105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:57.922125 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:57.923655 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:57.924111 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:57.924114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:57.924115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:57.924186 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:57.924524 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:57.924530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:57.924534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:57.924762 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:57.924862 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:57.925434 (+ 572us) server_negotiation.cc:299] Negotiation successful
0620 19:04:57.925713 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 19:04:57.931916 2212 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:04:57.934907 2212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:04:57.950639 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:57.944986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:44847 (local address 127.0.0.1:55380)
0620 19:04:57.945632 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:57.945646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:57.945852 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:57.946425 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:57.946428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:57.946443 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:57.946542 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:57.946546 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.947846 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.947850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:57.949041 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:57.949051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.950132 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.950135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:57.950255 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:57.950281 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 19:04:57.950391 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 19:04:57.950464 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":93,"threads_started":1}
I20260620 19:04:57.950680 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:57.945161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.1.154.131:44847)
0620 19:04:57.945763 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:57.945777 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:57.945801 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:04:57.946087 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:57.946095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:57.946268 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:57.946435 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:57.946623 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.947699 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.949173 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.949976 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.950079 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:57.950204 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:57.950433 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 19:04:57.950534 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":120,"threads_started":1}
I20260620 19:04:57.950935 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:57.944155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.1.154.130:37719)
0620 19:04:57.944590 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:57.944598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:57.944718 (+ 120us) server_negotiation.cc:407] Connection header received
0620 19:04:57.945354 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:57.945359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:57.945510 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:57.945671 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:57.946108 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.947308 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.948835 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.949678 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.949785 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:57.949926 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:57.950541 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 19:04:57.950628 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":88,"threads_started":1}
I20260620 19:04:57.951198 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:57.943945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:37719 (local address 127.0.0.1:48328)
0620 19:04:57.944569 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:57.944615 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:57.945211 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:57.945923 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:57.945926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:57.945938 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:57.946006 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:57.946010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.947465 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.947468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:57.948715 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:57.948724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.949991 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.949996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:57.950104 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:57.950139 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 19:04:57.950245 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 19:04:57.950385 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":70,"threads_started":1}
I20260620 19:04:57.951453 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:57.943994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:34953 (local address 127.0.0.1:53218)
0620 19:04:57.944911 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:57.944926 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:57.945054 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:57.945838 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:57.945844 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:57.945866 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:57.946002 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:57.946009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.948725 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.948728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:57.949486 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:57.949493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.950956 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.950959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:57.951124 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:57.951147 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:04:57.951218 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:04:57.951286 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":103,"threads_started":1}
I20260620 19:04:57.952512 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:57.944116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53218 (local address 127.1.154.129:34953)
0620 19:04:57.944754 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:57.944760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:57.945194 (+ 434us) server_negotiation.cc:407] Connection header received
0620 19:04:57.945278 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:57.945285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:57.945511 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:57.945676 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:57.947490 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.948601 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.949631 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:57.950465 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:57.950559 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:57.950684 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:57.952273 (+ 1589us) server_negotiation.cc:299] Negotiation successful
0620 19:04:57.952363 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":77,"threads_started":1}
I20260620 19:04:57.954072 2605 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd6b39e3cff4c51af8c6c1dd331ba0e (DEFAULT_TABLE table=test-table [id=c119a75099594652adc2bf5af56ff6c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:57.954053 2471 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd6b39e3cff4c51af8c6c1dd331ba0e (DEFAULT_TABLE table=test-table [id=c119a75099594652adc2bf5af56ff6c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:57.955092 2471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd6b39e3cff4c51af8c6c1dd331ba0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:57.955113 2605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd6b39e3cff4c51af8c6c1dd331ba0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:57.954950 2337 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd6b39e3cff4c51af8c6c1dd331ba0e (DEFAULT_TABLE table=test-table [id=c119a75099594652adc2bf5af56ff6c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:57.955792 2337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd6b39e3cff4c51af8c6c1dd331ba0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:57.960881 2688 tablet_bootstrap.cc:492] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f: Bootstrap starting.
I20260620 19:04:57.963304 2689 tablet_bootstrap.cc:492] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 135d318c5a84403682c25460a081763a: Bootstrap starting.
I20260620 19:04:57.963742 2688 tablet_bootstrap.cc:654] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:57.964515 2688 log.cc:824] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:57.965119 2690 tablet_bootstrap.cc:492] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 466c0286d9fc439399556a697f77b9ce: Bootstrap starting.
I20260620 19:04:57.965308 2689 tablet_bootstrap.cc:654] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 135d318c5a84403682c25460a081763a: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:57.966037 2689 log.cc:824] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 135d318c5a84403682c25460a081763a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:57.967373 2690 tablet_bootstrap.cc:654] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 466c0286d9fc439399556a697f77b9ce: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:57.968051 2689 tablet_bootstrap.cc:492] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 135d318c5a84403682c25460a081763a: No bootstrap required, opened a new log
I20260620 19:04:57.968209 2689 ts_tablet_manager.cc:1397] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 135d318c5a84403682c25460a081763a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:04:57.968173 2690 log.cc:824] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 466c0286d9fc439399556a697f77b9ce: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:57.969652 2688 tablet_bootstrap.cc:492] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f: No bootstrap required, opened a new log
I20260620 19:04:57.969833 2688 ts_tablet_manager.cc:1397] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260620 19:04:57.970297 2690 tablet_bootstrap.cc:492] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 466c0286d9fc439399556a697f77b9ce: No bootstrap required, opened a new log
I20260620 19:04:57.970492 2690 ts_tablet_manager.cc:1397] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 466c0286d9fc439399556a697f77b9ce: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:04:57.972834 2688 raft_consensus.cc:348] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5533734bb2143c3b5d6b102c32e355f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37719 } } peers { permanent_uuid: "135d318c5a84403682c25460a081763a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 34953 } } peers { permanent_uuid: "466c0286d9fc439399556a697f77b9ce" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44847 } }
I20260620 19:04:57.972915 2689 raft_consensus.cc:348] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 135d318c5a84403682c25460a081763a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5533734bb2143c3b5d6b102c32e355f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37719 } } peers { permanent_uuid: "135d318c5a84403682c25460a081763a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 34953 } } peers { permanent_uuid: "466c0286d9fc439399556a697f77b9ce" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44847 } }
I20260620 19:04:57.973150 2688 raft_consensus.cc:374] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:57.973172 2689 raft_consensus.cc:374] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 135d318c5a84403682c25460a081763a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:57.973205 2688 raft_consensus.cc:729] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5533734bb2143c3b5d6b102c32e355f, State: Initialized, Role: FOLLOWER
I20260620 19:04:57.973225 2689 raft_consensus.cc:729] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 135d318c5a84403682c25460a081763a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135d318c5a84403682c25460a081763a, State: Initialized, Role: FOLLOWER
I20260620 19:04:57.973711 2688 consensus_queue.cc:260] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5533734bb2143c3b5d6b102c32e355f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37719 } } peers { permanent_uuid: "135d318c5a84403682c25460a081763a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 34953 } } peers { permanent_uuid: "466c0286d9fc439399556a697f77b9ce" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44847 } }
I20260620 19:04:57.973747 2689 consensus_queue.cc:260] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 135d318c5a84403682c25460a081763a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5533734bb2143c3b5d6b102c32e355f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37719 } } peers { permanent_uuid: "135d318c5a84403682c25460a081763a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 34953 } } peers { permanent_uuid: "466c0286d9fc439399556a697f77b9ce" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44847 } }
I20260620 19:04:57.974438 2537 heartbeater.cc:503] Master 127.1.154.190:40905 was elected leader, sending a full tablet report...
I20260620 19:04:57.974848 2689 ts_tablet_manager.cc:1428] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 135d318c5a84403682c25460a081763a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 19:04:57.974647 2690 raft_consensus.cc:348] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 466c0286d9fc439399556a697f77b9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5533734bb2143c3b5d6b102c32e355f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37719 } } peers { permanent_uuid: "135d318c5a84403682c25460a081763a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 34953 } } peers { permanent_uuid: "466c0286d9fc439399556a697f77b9ce" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44847 } }
I20260620 19:04:57.974943 2690 raft_consensus.cc:374] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 466c0286d9fc439399556a697f77b9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:57.975001 2690 raft_consensus.cc:729] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 466c0286d9fc439399556a697f77b9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 466c0286d9fc439399556a697f77b9ce, State: Initialized, Role: FOLLOWER
I20260620 19:04:57.975207 2403 heartbeater.cc:503] Master 127.1.154.190:40905 was elected leader, sending a full tablet report...
I20260620 19:04:57.975454 2690 consensus_queue.cc:260] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 466c0286d9fc439399556a697f77b9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5533734bb2143c3b5d6b102c32e355f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37719 } } peers { permanent_uuid: "135d318c5a84403682c25460a081763a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 34953 } } peers { permanent_uuid: "466c0286d9fc439399556a697f77b9ce" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44847 } }
I20260620 19:04:57.975929 2688 ts_tablet_manager.cc:1428] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:04:57.976377 2690 ts_tablet_manager.cc:1428] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 466c0286d9fc439399556a697f77b9ce: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:04:57.976428 2671 heartbeater.cc:503] Master 127.1.154.190:40905 was elected leader, sending a full tablet report...
I20260620 19:04:58.095300 2694 raft_consensus.cc:482] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:04:58.095590 2694 raft_consensus.cc:504] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5533734bb2143c3b5d6b102c32e355f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37719 } } peers { permanent_uuid: "135d318c5a84403682c25460a081763a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 34953 } } peers { permanent_uuid: "466c0286d9fc439399556a697f77b9ce" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44847 } }
I20260620 19:04:58.096808 2694 leader_election.cc:288] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 135d318c5a84403682c25460a081763a (127.1.154.129:34953), 466c0286d9fc439399556a697f77b9ce (127.1.154.131:44847)
I20260620 19:04:58.102691 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:58.097530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:45619 (local address 127.1.154.131:44847)
0620 19:04:58.097729 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:58.097734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:58.098073 (+ 339us) server_negotiation.cc:407] Connection header received
0620 19:04:58.098209 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:58.098215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:58.098279 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:58.098382 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:58.098787 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:58.099408 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:58.101348 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:58.102015 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:58.102150 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:58.102206 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:58.102473 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 19:04:58.102549 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 19:04:58.102957 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:58.097136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:34953 (local address 127.1.154.130:49363)
0620 19:04:58.097283 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:58.097309 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:58.097409 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:58.097971 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:58.097973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:58.097991 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:58.098183 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:58.098192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:58.099519 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:58.099523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:58.101275 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:58.101286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:58.102557 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:58.102561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:58.102664 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:58.102686 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:04:58.102762 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:04:58.102845 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":125}
I20260620 19:04:58.102983 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:58.097136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:44847 (local address 127.1.154.130:45619)
0620 19:04:58.097941 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:58.097959 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:58.098086 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:58.098426 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:58.098429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:58.098503 (+ 74us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:58.098642 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:58.098650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:58.099519 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:58.099523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:58.101195 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:58.101210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:58.102158 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:58.102163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:58.102282 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:58.102314 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 19:04:58.102494 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 19:04:58.102585 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":422,"thread_start_us":163,"threads_started":1}
I20260620 19:04:58.103341 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:58.097372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:49363 (local address 127.1.154.129:34953)
0620 19:04:58.097565 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:58.097572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:58.097624 (+ 52us) server_negotiation.cc:407] Connection header received
0620 19:04:58.097710 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:58.097714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:58.097784 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:58.097915 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:58.098394 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:58.099302 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:58.101470 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:58.102140 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:58.102287 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:58.102340 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:58.103154 (+ 814us) server_negotiation.cc:299] Negotiation successful
0620 19:04:58.103250 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 19:04:58.103775 2625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd6b39e3cff4c51af8c6c1dd331ba0e" candidate_uuid: "f5533734bb2143c3b5d6b102c32e355f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "466c0286d9fc439399556a697f77b9ce" is_pre_election: true
I20260620 19:04:58.104110 2625 raft_consensus.cc:2454] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 466c0286d9fc439399556a697f77b9ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5533734bb2143c3b5d6b102c32e355f in term 0.
I20260620 19:04:58.104570 2425 leader_election.cc:302] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 466c0286d9fc439399556a697f77b9ce, f5533734bb2143c3b5d6b102c32e355f; no voters:
I20260620 19:04:58.104861 2694 raft_consensus.cc:2781] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:04:58.104848 2356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd6b39e3cff4c51af8c6c1dd331ba0e" candidate_uuid: "f5533734bb2143c3b5d6b102c32e355f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "135d318c5a84403682c25460a081763a" is_pre_election: true
I20260620 19:04:58.104951 2694 raft_consensus.cc:482] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:04:58.104996 2694 raft_consensus.cc:3037] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:58.105110 2356 raft_consensus.cc:2454] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 135d318c5a84403682c25460a081763a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5533734bb2143c3b5d6b102c32e355f in term 0.
I20260620 19:04:58.106191 2694 raft_consensus.cc:504] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5533734bb2143c3b5d6b102c32e355f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37719 } } peers { permanent_uuid: "135d318c5a84403682c25460a081763a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 34953 } } peers { permanent_uuid: "466c0286d9fc439399556a697f77b9ce" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44847 } }
I20260620 19:04:58.106877 2694 leader_election.cc:288] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [CANDIDATE]: Term 1 election: Requested vote from peers 135d318c5a84403682c25460a081763a (127.1.154.129:34953), 466c0286d9fc439399556a697f77b9ce (127.1.154.131:44847)
I20260620 19:04:58.107188 2356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd6b39e3cff4c51af8c6c1dd331ba0e" candidate_uuid: "f5533734bb2143c3b5d6b102c32e355f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "135d318c5a84403682c25460a081763a"
I20260620 19:04:58.107281 2356 raft_consensus.cc:3037] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 135d318c5a84403682c25460a081763a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:58.107582 2625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd6b39e3cff4c51af8c6c1dd331ba0e" candidate_uuid: "f5533734bb2143c3b5d6b102c32e355f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "466c0286d9fc439399556a697f77b9ce"
I20260620 19:04:58.107689 2625 raft_consensus.cc:3037] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 466c0286d9fc439399556a697f77b9ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:58.108582 2356 raft_consensus.cc:2454] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 135d318c5a84403682c25460a081763a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5533734bb2143c3b5d6b102c32e355f in term 1.
I20260620 19:04:58.108964 2426 leader_election.cc:302] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135d318c5a84403682c25460a081763a, f5533734bb2143c3b5d6b102c32e355f; no voters:
I20260620 19:04:58.108986 2625 raft_consensus.cc:2454] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 466c0286d9fc439399556a697f77b9ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5533734bb2143c3b5d6b102c32e355f in term 1.
I20260620 19:04:58.109163 2694 raft_consensus.cc:2781] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:58.109422 2694 raft_consensus.cc:686] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [term 1 LEADER]: Becoming Leader. State: Replica: f5533734bb2143c3b5d6b102c32e355f, State: Running, Role: LEADER
I20260620 19:04:58.109858 2694 consensus_queue.cc:237] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5533734bb2143c3b5d6b102c32e355f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37719 } } peers { permanent_uuid: "135d318c5a84403682c25460a081763a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 34953 } } peers { permanent_uuid: "466c0286d9fc439399556a697f77b9ce" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44847 } }
I20260620 19:04:58.112937 2212 catalog_manager.cc:5495] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f reported cstate change: term changed from 0 to 1, leader changed from <none> to f5533734bb2143c3b5d6b102c32e355f (127.1.154.130). New cstate: current_term: 1 leader_uuid: "f5533734bb2143c3b5d6b102c32e355f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5533734bb2143c3b5d6b102c32e355f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "135d318c5a84403682c25460a081763a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 34953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "466c0286d9fc439399556a697f77b9ce" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44847 } health_report { overall_health: UNKNOWN } } }
W20260620 19:04:58.133502 2672 tablet.cc:2369] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 466c0286d9fc439399556a697f77b9ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 19:04:58 dist-test-slave-6wrf krb5kdc[2167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.129@KRBTEST.COM
I20260620 19:04:58.152364 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:58.142429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.1.154.129:34953)
0620 19:04:58.142591 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:58.142597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:58.142730 (+ 133us) server_negotiation.cc:407] Connection header received
0620 19:04:58.142920 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:58.142924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:58.142984 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:58.143087 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:58.143962 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:58.144491 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:58.145374 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:58.145674 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:58.145805 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:58.148541 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:58.148553 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:58.148572 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:58.148605 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:58.150511 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:58.151000 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:58.151006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:58.151007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:58.151097 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:58.151422 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:58.151425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:58.151427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:58.151681 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:58.151888 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:58.152070 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 19:04:58.152191 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 19:04:58.162416 2538 tablet.cc:2369] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P f5533734bb2143c3b5d6b102c32e355f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 19:04:58 dist-test-slave-6wrf krb5kdc[2167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260620 19:04:58.173448 2404 tablet.cc:2369] T 8fd6b39e3cff4c51af8c6c1dd331ba0e P 135d318c5a84403682c25460a081763a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 19:04:58 dist-test-slave-6wrf krb5kdc[2167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.154.129@KRBTEST.COM
I20260620 19:04:58.182691 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:58.172412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.1.154.129:34953)
0620 19:04:58.172591 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:58.172594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:58.172676 (+ 82us) server_negotiation.cc:407] Connection header received
0620 19:04:58.172905 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:58.172909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:58.173123 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:58.173215 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:58.174030 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:58.174537 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:58.175476 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:58.175781 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:58.175935 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:58.178832 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:58.178836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:58.178839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:58.178873 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:58.180698 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:58.181200 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:58.181203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:58.181204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:58.181297 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:58.181718 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:58.181723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:58.181725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:58.181928 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:58.182043 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:58.182350 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 19:04:58.182460 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 19:04:58 dist-test-slave-6wrf krb5kdc[2167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:04:58 dist-test-slave-6wrf krb5kdc[2167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.129@KRBTEST.COM
I20260620 19:04:58.210702 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:58.199980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53246 (local address 127.1.154.129:34953)
0620 19:04:58.200290 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:58.200296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:58.200652 (+ 356us) server_negotiation.cc:407] Connection header received
0620 19:04:58.201006 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:58.201010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:58.201082 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:58.201192 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:58.202211 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:58.202934 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:58.204077 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:58.204330 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:58.204456 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:58.207126 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:58.207130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:58.207132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:58.207162 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:58.208996 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:58.209466 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:58.209474 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:58.209476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:58.209546 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:58.209837 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:58.209842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:58.209844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:58.210022 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:58.210159 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:58.210396 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 19:04:58.210523 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
W20260620 19:04:58.212707 2317 tablet_service.cc:2908] Rejecting scan request for tablet 8fd6b39e3cff4c51af8c6c1dd331ba0e: Uninitialized: clean time has not yet been initialized
I20260620 19:04:58.215225 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 2274
I20260620 19:04:58.220952 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 2408
I20260620 19:04:58.227015 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 2542
I20260620 19:04:58.233188 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 2180
2026-06-20T19:04:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2514 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:58 dist-test-slave-6wrf krb5kdc[2712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:58 dist-test-slave-6wrf krb5kdc[2712](info): set up 2 sockets
Jun 20 19:04:58 dist-test-slave-6wrf krb5kdc[2712](info): commencing operation
krb5kdc: starting...
W20260620 19:04:59.313560 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:04:59 dist-test-slave-6wrf krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:04:59.448477 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:44331
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:44331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:59.562999 2725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:59.563279 2725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:59.563321 2725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:59.567458 2725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:59.567514 2725 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:04:59.567533 2725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:59.567550 2725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:59.567605 2725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:59.567641 2725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:59.573033 2725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:44331
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:44331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:59.574177 2725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:59.575278 2725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:59.583981 2733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:59.584049 2731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:59.584086 2730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:59.584769 2725 server_base.cc:1034] running on GCE node
I20260620 19:04:59.585461 2725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:59.586666 2725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:59.588017 2725 hybrid_clock.cc:648] HybridClock initialized: now 1781982299587986 us; error 74 us; skew 500 ppm
Jun 20 19:04:59 dist-test-slave-6wrf krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982299, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:59.594580 2725 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:04:59.595603 2725 webserver.cc:460] Webserver started at http://127.1.154.190:35889/ using document root <none> and password file <none>
I20260620 19:04:59.596225 2725 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:59.596321 2725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:59.596550 2725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:59.598706 2725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "9a0aeaf2c1f04b86aaba045c477edde4"
format_stamp: "Formatted at 2026-06-20 19:04:59 on dist-test-slave-6wrf"
server_key: "ad7c98be5d40a247126578f07bb1bd29"
server_key_iv: "c528dcf90704ef2a6a2e82df2824087e"
server_key_version: "encryptionkey@0"
I20260620 19:04:59.599260 2725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "9a0aeaf2c1f04b86aaba045c477edde4"
format_stamp: "Formatted at 2026-06-20 19:04:59 on dist-test-slave-6wrf"
server_key: "ad7c98be5d40a247126578f07bb1bd29"
server_key_iv: "c528dcf90704ef2a6a2e82df2824087e"
server_key_version: "encryptionkey@0"
I20260620 19:04:59.603448 2725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:04:59.606199 2740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:59.607687 2725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:04:59.607846 2725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "9a0aeaf2c1f04b86aaba045c477edde4"
format_stamp: "Formatted at 2026-06-20 19:04:59 on dist-test-slave-6wrf"
server_key: "ad7c98be5d40a247126578f07bb1bd29"
server_key_iv: "c528dcf90704ef2a6a2e82df2824087e"
server_key_version: "encryptionkey@0"
I20260620 19:04:59.607980 2725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:59.631183 2725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:59.634626 2725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:59.634847 2725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:59.644758 2725 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:44331
I20260620 19:04:59.644829 2802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:44331 every 8 connection(s)
I20260620 19:04:59.645843 2725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:59.646657 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 2725
I20260620 19:04:59.646777 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:04:59.647032 1642 external_mini_cluster.cc:1428] Setting key 8756b294776a886d384f52da519b9703
I20260620 19:04:59.649259 2803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:59.656309 2803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4: Bootstrap starting.
Jun 20 19:04:59 dist-test-slave-6wrf krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:04:59.659536 2803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:59.660392 2803 log.cc:824] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:59.663260 2803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4: No bootstrap required, opened a new log
I20260620 19:04:59.666951 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:59.648933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50762 (local address 127.1.154.190:44331)
0620 19:04:59.649654 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:59.649688 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:59.649741 (+ 53us) server_negotiation.cc:407] Connection header received
0620 19:04:59.650749 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:59.650792 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:59.651313 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:59.651784 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:59.652795 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:59.654094 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:59.655419 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:59.655995 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:59.656159 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:59.660231 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:59.660241 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:59.660261 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:59.660303 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:59.663112 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:59.663988 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:59.663995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:59.664002 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:59.664175 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:59.664574 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:59.664577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:59.664579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:59.665104 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:59.665294 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:59.665662 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 19:04:59.665972 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":172,"threads_started":1}
I20260620 19:04:59.667590 2803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0aeaf2c1f04b86aaba045c477edde4" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 44331 } }
I20260620 19:04:59.667860 2803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:59.667908 2803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a0aeaf2c1f04b86aaba045c477edde4, State: Initialized, Role: FOLLOWER
I20260620 19:04:59.668352 2803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0aeaf2c1f04b86aaba045c477edde4" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 44331 } }
I20260620 19:04:59.668473 2803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:59.668553 2803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:59.668649 2803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:59.669775 2803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0aeaf2c1f04b86aaba045c477edde4" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 44331 } }
I20260620 19:04:59.670113 2803 leader_election.cc:302] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a0aeaf2c1f04b86aaba045c477edde4; no voters:
I20260620 19:04:59.670431 2803 leader_election.cc:288] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:59.670616 2808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:59.671105 2808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [term 1 LEADER]: Becoming Leader. State: Replica: 9a0aeaf2c1f04b86aaba045c477edde4, State: Running, Role: LEADER
I20260620 19:04:59.671398 2808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0aeaf2c1f04b86aaba045c477edde4" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 44331 } }
I20260620 19:04:59.671782 2803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:59.673130 2809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a0aeaf2c1f04b86aaba045c477edde4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0aeaf2c1f04b86aaba045c477edde4" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 44331 } } }
I20260620 19:04:59.673169 2810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a0aeaf2c1f04b86aaba045c477edde4. Latest consensus state: current_term: 1 leader_uuid: "9a0aeaf2c1f04b86aaba045c477edde4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0aeaf2c1f04b86aaba045c477edde4" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 44331 } } }
I20260620 19:04:59.673250 2809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:59.673257 2810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:59.673807 2820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:59.676582 2820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:59.683876 2820 catalog_manager.cc:1347] Generated new cluster ID: 846a505b239948c0803dc2088ddfeb1c
I20260620 19:04:59.683969 2820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:59.699003 2820 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:59.700263 2820 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:59.706990 2820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4: Generated new TSK 0
I20260620 19:04:59.707670 2820 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:04:59.750705 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:44331
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:59.867789 2829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:59.868095 2829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:59.868170 2829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:59.872467 2829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:59.872551 2829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:59.872617 2829 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:04:59.872690 2829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:04:59.878147 2829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.154.190:44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:59.879212 2829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:59.880270 2829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:59.890298 2834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:59.890326 2829 server_base.cc:1034] running on GCE node
W20260620 19:04:59.890345 2837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:59.890326 2835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:59.890990 2829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:59.891654 2829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:59.892866 2829 hybrid_clock.cc:648] HybridClock initialized: now 1781982299892831 us; error 47 us; skew 500 ppm
Jun 20 19:04:59 dist-test-slave-6wrf krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982299, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:59.896411 2829 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:04:59.897322 2829 webserver.cc:460] Webserver started at http://127.1.154.129:44559/ using document root <none> and password file <none>
I20260620 19:04:59.897953 2829 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:59.898046 2829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:59.898275 2829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:59.900367 2829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "ff042333a51246748ca1eb7bf3273cf2"
format_stamp: "Formatted at 2026-06-20 19:04:59 on dist-test-slave-6wrf"
server_key: "f7b0b42b6f16289ed9999cbaae90955c"
server_key_iv: "43f8c37aa8b49b79563ee26ab511a3f0"
server_key_version: "encryptionkey@0"
I20260620 19:04:59.900969 2829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "ff042333a51246748ca1eb7bf3273cf2"
format_stamp: "Formatted at 2026-06-20 19:04:59 on dist-test-slave-6wrf"
server_key: "f7b0b42b6f16289ed9999cbaae90955c"
server_key_iv: "43f8c37aa8b49b79563ee26ab511a3f0"
server_key_version: "encryptionkey@0"
I20260620 19:04:59.906606 2829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 19:04:59.910341 2844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:59.911911 2829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:59.912066 2829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "ff042333a51246748ca1eb7bf3273cf2"
format_stamp: "Formatted at 2026-06-20 19:04:59 on dist-test-slave-6wrf"
server_key: "f7b0b42b6f16289ed9999cbaae90955c"
server_key_iv: "43f8c37aa8b49b79563ee26ab511a3f0"
server_key_version: "encryptionkey@0"
I20260620 19:04:59.912209 2829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:59.928812 2829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:59.932140 2829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:59.932353 2829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:59.934656 2829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:59.934762 2829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:59.934831 2829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:59.934859 2829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:04:59 dist-test-slave-6wrf krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982299, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:04:59.946730 2829 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:38143
I20260620 19:04:59.946769 2959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:38143 every 8 connection(s)
I20260620 19:04:59.947875 2829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:04:59.950397 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 2829
I20260620 19:04:59.950525 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:04:59.950838 1642 external_mini_cluster.cc:1428] Setting key dd9a9e01453c02b4f3b3b69084babf76
I20260620 19:04:59.952054 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:59.935541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:37747 (local address 127.1.154.190:44331)
0620 19:04:59.935744 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:59.935750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:59.937285 (+ 1535us) server_negotiation.cc:407] Connection header received
0620 19:04:59.938723 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:59.938727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:59.938809 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:59.938911 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:59.940572 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:59.941347 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:59.943192 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:59.943492 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:59.943573 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:59.947217 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:59.947222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:59.947225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:59.947255 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:59.948955 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:59.949918 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:59.949921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:59.949922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:59.950006 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:59.950671 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:59.950676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:59.950678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:59.951030 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:59.951154 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:59.951742 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 19:04:59.951891 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 19:04:59.952960 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:59.936404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:44331 (local address 127.1.154.129:37747)
0620 19:04:59.937092 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:59.937159 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:59.938455 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:59.939105 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:59.939115 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:59.939655 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:59.940395 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:59.940417 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:59.941473 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:59.941479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:59.943042 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:59.943057 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:59.943665 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:59.943672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:59.943916 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:59.944601 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:59.944641 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:59.947042 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:59.949170 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:59.949178 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:59.949196 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:59.949740 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:59.950210 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:59.950216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:59.950218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:59.950420 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:59.951172 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:59.951180 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:59.951428 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 19:04:59.951656 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 19:04:59.951970 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":123,"threads_started":1}
I20260620 19:04:59.954559 2960 heartbeater.cc:347] Connected to a master server at 127.1.154.190:44331
I20260620 19:04:59.954777 2960 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:59.955262 2960 heartbeater.cc:511] Master 127.1.154.190:44331 requested a full tablet report, sending...
I20260620 19:04:59.956874 2757 ts_manager.cc:194] Registered new tserver with Master: ff042333a51246748ca1eb7bf3273cf2 (127.1.154.129:38143)
I20260620 19:04:59.958245 2757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:37747
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
I20260620 19:04:59.967175 2967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:59.958053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:44331 (local address 127.1.154.129:56191)
0620 19:04:59.958379 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:59.958396 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:59.958500 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:59.959052 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:59.959057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:59.959324 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:59.959493 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:59.959505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:59.960503 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:59.960508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:59.961476 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:59.961486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:59.962220 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:59.962226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:59.962362 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:59.962912 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:59.962930 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:59.963339 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:59.965273 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:59.965276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:59.965277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:59.965463 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:59.966102 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:59.966109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:59.966123 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:59.966241 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:59.966709 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:59.966711 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:59.966791 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 19:04:59.966889 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 19:04:59.967005 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":109,"threads_started":1}
I20260620 19:04:59.967332 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:59.958073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:56191 (local address 127.1.154.190:44331)
0620 19:04:59.958815 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:59.958818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:59.958851 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:04:59.958888 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:59.958890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:59.958929 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:59.959014 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:59.959638 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:59.960362 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:59.961650 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:59.962075 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:59.962196 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:59.963470 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:59.963473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:59.963475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:59.963495 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:59.965142 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:59.965568 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:59.965571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:59.965572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:59.965750 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:59.966357 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:59.966360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:59.966361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:59.966588 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:59.966695 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:59.966971 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 19:04:59.967111 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:04:59.985877 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:44331
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:00.114709 2969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:00.115021 2969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:00.115126 2969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:00.119403 2969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:00.119506 2969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:00.119587 2969 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:05:00.119683 2969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:00.125221 2969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.154.190:44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:00.126416 2969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:00.127686 2969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:00.140045 2977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:00.140045 2975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:00.140053 2974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:00.140434 2969 server_base.cc:1034] running on GCE node
I20260620 19:05:00.141043 2969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:00.141870 2969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:00.143121 2969 hybrid_clock.cc:648] HybridClock initialized: now 1781982300143036 us; error 106 us; skew 500 ppm
Jun 20 19:05:00 dist-test-slave-6wrf krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982300, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:00.146502 2969 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:05:00.147425 2969 webserver.cc:460] Webserver started at http://127.1.154.130:35923/ using document root <none> and password file <none>
I20260620 19:05:00.148036 2969 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:00.148130 2969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:00.148360 2969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:00.151377 2969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "395b12ffc77546468d7977123d6a58a3"
format_stamp: "Formatted at 2026-06-20 19:05:00 on dist-test-slave-6wrf"
server_key: "88d65f700c2ce4243eac0ef0b37d7c1e"
server_key_iv: "5221220f966bd73ac996106a94321ccd"
server_key_version: "encryptionkey@0"
I20260620 19:05:00.152112 2969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "395b12ffc77546468d7977123d6a58a3"
format_stamp: "Formatted at 2026-06-20 19:05:00 on dist-test-slave-6wrf"
server_key: "88d65f700c2ce4243eac0ef0b37d7c1e"
server_key_iv: "5221220f966bd73ac996106a94321ccd"
server_key_version: "encryptionkey@0"
I20260620 19:05:00.157101 2969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 19:05:00.160164 2984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:00.161983 2969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 19:05:00.162153 2969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "395b12ffc77546468d7977123d6a58a3"
format_stamp: "Formatted at 2026-06-20 19:05:00 on dist-test-slave-6wrf"
server_key: "88d65f700c2ce4243eac0ef0b37d7c1e"
server_key_iv: "5221220f966bd73ac996106a94321ccd"
server_key_version: "encryptionkey@0"
I20260620 19:05:00.162287 2969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:00.177346 2969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:00.181458 2969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:00.181768 2969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:00.183735 2969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:00.183823 2969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:00.184073 2969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:00.184132 2969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:05:00 dist-test-slave-6wrf krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982300, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:00.195984 2969 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:39525
I20260620 19:05:00.196303 3099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:39525 every 8 connection(s)
I20260620 19:05:00.197045 2969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:00.199488 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.184695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:53015 (local address 127.1.154.190:44331)
0620 19:05:00.184861 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:00.184868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:00.185749 (+ 881us) server_negotiation.cc:407] Connection header received
0620 19:05:00.186700 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:00.186704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:00.186757 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:00.186835 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:00.188150 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.188989 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.190853 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.191273 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.191420 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.195228 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:00.195233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:00.195235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:00.195263 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:00.196988 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:00.197567 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:00.197570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:00.197572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:00.197693 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:00.198124 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:00.198127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:00.198128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:00.198315 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:00.198395 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:00.199077 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 19:05:00.199270 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:05:00.199936 2993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.185072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:44331 (local address 127.1.154.130:53015)
0620 19:05:00.185556 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:00.185636 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:00.186501 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:00.186991 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:00.186997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:00.187446 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:00.187968 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.187981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.189243 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.189248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.190681 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.190702 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.191500 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.191506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.191768 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.192887 (+ 1119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:00.192912 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:00.195037 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:00.197121 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:00.197129 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:00.197146 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:00.197455 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:00.197818 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:00.197822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:00.197825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:00.198010 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:00.198416 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:00.198421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:00.198652 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 19:05:00.198803 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 19:05:00.199059 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":164,"threads_started":1}
I20260620 19:05:00.201438 3100 heartbeater.cc:347] Connected to a master server at 127.1.154.190:44331
I20260620 19:05:00.201647 3100 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:00.202286 3100 heartbeater.cc:511] Master 127.1.154.190:44331 requested a full tablet report, sending...
I20260620 19:05:00.203569 2756 ts_manager.cc:194] Registered new tserver with Master: 395b12ffc77546468d7977123d6a58a3 (127.1.154.130:39525)
I20260620 19:05:00.203816 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 2969
I20260620 19:05:00.204085 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:05:00.204326 2756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:53015
I20260620 19:05:00.204484 1642 external_mini_cluster.cc:1428] Setting key a2fc755a2606ce0e148624da99575634
I20260620 19:05:00.214215 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.203867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:46411 (local address 127.1.154.190:44331)
0620 19:05:00.203979 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:00.203983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:00.204424 (+ 441us) server_negotiation.cc:407] Connection header received
0620 19:05:00.204602 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:00.204606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:00.204681 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:00.204786 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:00.205542 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.206355 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.207474 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.207903 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.208039 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.209557 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:00.209561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:00.209563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:00.209587 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:00.211837 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:00.212703 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:00.212706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:00.212707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:00.212795 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:00.213229 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:00.213232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:00.213233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:00.213451 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:00.213540 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:00.213945 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 19:05:00.214083 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:05:00.214521 3106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.203881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:44331 (local address 127.1.154.130:46411)
0620 19:05:00.204301 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:00.204317 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:00.204427 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:00.204939 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:00.204943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:00.205223 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:00.205412 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.205420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.206586 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.206591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.207338 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.207347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.208042 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.208045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.208127 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.208857 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:00.208883 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:00.209374 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:00.212000 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:00.212006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:00.212009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:00.212548 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:00.212926 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:00.212931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:00.212936 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:00.213090 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:00.213568 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:00.213574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:00.213750 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 19:05:00.213885 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 19:05:00.214127 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":3,"thread_start_us":225,"threads_started":1}
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:05:00.237186 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:44331
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:00.349486 3109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:00.349848 3109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:00.349921 3109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:00.354082 3109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:00.354163 3109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:00.354229 3109 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:05:00.354315 3109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:00.359784 3109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.154.190:44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:00.360931 3109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:00.361995 3109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:00.372215 3114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:00.372561 3115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:00.372614 3117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:00.372818 3109 server_base.cc:1034] running on GCE node
I20260620 19:05:00.373229 3109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:00.373982 3109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:00.375289 3109 hybrid_clock.cc:648] HybridClock initialized: now 1781982300375254 us; error 72 us; skew 500 ppm
Jun 20 19:05:00 dist-test-slave-6wrf krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982300, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:00.378309 3109 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:05:00.379247 3109 webserver.cc:460] Webserver started at http://127.1.154.131:44523/ using document root <none> and password file <none>
I20260620 19:05:00.379841 3109 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:00.379932 3109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:00.380168 3109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:00.382381 3109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "43492d48a9944626b60b2cbc95cfc5d0"
format_stamp: "Formatted at 2026-06-20 19:05:00 on dist-test-slave-6wrf"
server_key: "de70ee12504b9fb0daddbb5298ab0c97"
server_key_iv: "4a9eb6be5557ac8fb4481e324ea4125c"
server_key_version: "encryptionkey@0"
I20260620 19:05:00.382951 3109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "43492d48a9944626b60b2cbc95cfc5d0"
format_stamp: "Formatted at 2026-06-20 19:05:00 on dist-test-slave-6wrf"
server_key: "de70ee12504b9fb0daddbb5298ab0c97"
server_key_iv: "4a9eb6be5557ac8fb4481e324ea4125c"
server_key_version: "encryptionkey@0"
I20260620 19:05:00.387238 3109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:05:00.390043 3124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:00.391717 3109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:05:00.391870 3109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "43492d48a9944626b60b2cbc95cfc5d0"
format_stamp: "Formatted at 2026-06-20 19:05:00 on dist-test-slave-6wrf"
server_key: "de70ee12504b9fb0daddbb5298ab0c97"
server_key_iv: "4a9eb6be5557ac8fb4481e324ea4125c"
server_key_version: "encryptionkey@0"
I20260620 19:05:00.391991 3109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:00.410938 3109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:00.415295 3109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:00.415556 3109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:00.418000 3109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:00.418128 3109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:00.418231 3109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:00.418269 3109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:05:00 dist-test-slave-6wrf krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982300, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:00.432528 3109 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:36395
I20260620 19:05:00.432576 3239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:36395 every 8 connection(s)
I20260620 19:05:00.433785 3109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:00.434067 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.419209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:51111 (local address 127.1.154.190:44331)
0620 19:05:00.419362 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:00.419365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:00.420849 (+ 1484us) server_negotiation.cc:407] Connection header received
0620 19:05:00.422518 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:00.422522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:00.422604 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:00.422719 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:00.423979 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.424588 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.426139 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.426427 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.426540 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.430096 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:00.430109 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:00.430111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:00.430146 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:00.431643 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:00.432183 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:00.432186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:00.432188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:00.432300 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:00.432715 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:00.432720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:00.432725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:00.432949 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:00.433028 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:00.433768 (+ 740us) server_negotiation.cc:299] Negotiation successful
0620 19:05:00.433903 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 19:05:00.434877 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 3109
I20260620 19:05:00.435005 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:05:00.435314 1642 external_mini_cluster.cc:1428] Setting key f45ac4387a61b59af0f79178b28126bd
I20260620 19:05:00.435091 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.419865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:44331 (local address 127.1.154.131:51111)
0620 19:05:00.420552 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:00.420609 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:00.422107 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:00.422915 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:00.422922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:00.423319 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:00.423825 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.423839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.424712 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.424718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.426014 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.426023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.426577 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.426582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.426792 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.427447 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:00.427463 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:00.429910 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:00.431773 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:00.431779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:00.431805 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:00.432073 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:00.432381 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:00.432385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:00.432387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:00.432597 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:00.433074 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:00.433080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:00.433401 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 19:05:00.433663 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 19:05:00.434029 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":169,"threads_started":1}
I20260620 19:05:00.437040 3240 heartbeater.cc:347] Connected to a master server at 127.1.154.190:44331
I20260620 19:05:00.437213 3240 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:00.437850 3240 heartbeater.cc:511] Master 127.1.154.190:44331 requested a full tablet report, sending...
I20260620 19:05:00.438972 2756 ts_manager.cc:194] Registered new tserver with Master: 43492d48a9944626b60b2cbc95cfc5d0 (127.1.154.131:36395)
I20260620 19:05:00.439913 2756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:51111
I20260620 19:05:00.447333 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.439235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:56369 (local address 127.1.154.190:44331)
0620 19:05:00.439355 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:00.439358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:00.439629 (+ 271us) server_negotiation.cc:407] Connection header received
0620 19:05:00.439739 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:00.439742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:00.439779 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:00.439849 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:00.440425 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.441674 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.442721 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.443042 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.443174 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.444310 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:00.444313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:00.444315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:00.444335 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:00.445869 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:00.446321 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:00.446323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:00.446325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:00.446402 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:00.446693 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:00.446695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:00.446696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:00.446848 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:00.446922 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:00.447139 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 19:05:00.447223 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:05:00.447368 3246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.439151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:44331 (local address 127.1.154.131:56369)
0620 19:05:00.439498 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:00.439529 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:00.439644 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:00.439896 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:00.439899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:00.440134 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:00.440289 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.440297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.441873 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.441879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.442536 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.442550 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.443225 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.443228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.443314 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.443814 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:00.443825 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:00.444165 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:00.445985 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:00.445988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:00.445989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:00.446227 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:00.446515 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:00.446517 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:00.446518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:00.446608 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:00.446933 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:00.446935 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:00.447018 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 19:05:00.447102 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:05:00.447210 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":9,"thread_start_us":109,"threads_started":1}
I20260620 19:05:00.449290 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:05:00 dist-test-slave-6wrf krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982300, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:05:00 dist-test-slave-6wrf krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982300, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:00.477274 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.467373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.1.154.190:44331)
0620 19:05:00.467525 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:00.467530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:00.467544 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:05:00.467594 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:00.467597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:00.467651 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:00.467740 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:00.468717 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.469522 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.470835 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.471201 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.471318 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.474010 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:00.474016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:00.474024 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:00.474047 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:00.475735 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:00.476147 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:00.476151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:00.476152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:00.476223 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:00.476465 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:00.476468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:00.476469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:00.476658 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:00.476770 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:00.476973 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 19:05:00.477060 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:05:00.481102 2756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:05:00.484210 2756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:00.499131 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.493003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.1.154.129:38143)
0620 19:05:00.494140 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:00.494148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:00.494169 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:05:00.494243 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:00.494250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:00.494464 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:00.494611 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:00.494916 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.496067 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.497663 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.498421 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.498538 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.498685 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:00.498863 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 19:05:00.498975 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1028,"thread_start_us":74,"threads_started":1}
I20260620 19:05:00.499166 3258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.492899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:36395 (local address 127.0.0.1:37338)
0620 19:05:00.493711 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:00.493727 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:00.493872 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:00.494765 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:00.494768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:00.494792 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:00.494900 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.494906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.496368 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.496371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.497822 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.497832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.498680 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.498683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.498754 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.498781 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:05:00.498827 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:05:00.498981 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":26,"spinlock_wait_cycles":7808,"thread_start_us":113,"threads_started":1}
I20260620 19:05:00.499377 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.492919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:39525 (local address 127.0.0.1:36134)
0620 19:05:00.494069 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:00.494082 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:00.494155 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:00.495058 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:00.495060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:00.495072 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:00.495148 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.495152 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.496127 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.496131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.497785 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.497795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.498979 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.498982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.499077 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.499099 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:05:00.499171 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:05:00.499250 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"mutex_wait_us":122,"thread_start_us":103,"threads_started":1}
I20260620 19:05:00.499166 3259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.492865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:38143 (local address 127.0.0.1:37392)
0620 19:05:00.493668 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:00.493717 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:00.493872 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:00.494626 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:00.494632 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:00.494654 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:00.494788 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.494794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.496194 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.496197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.497525 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.497535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.498549 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.498553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.498671 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.498712 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 19:05:00.498822 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 19:05:00.498981 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":131,"spinlock_wait_cycles":2304,"thread_start_us":83,"threads_started":1}
I20260620 19:05:00.499845 3261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.493656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.1.154.130:39525)
0620 19:05:00.494506 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:00.494517 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:00.494534 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:05:00.494625 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:00.494630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:00.494822 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:00.494952 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:00.495256 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.496014 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.497913 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.498654 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.498762 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.498920 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:00.499572 (+ 652us) server_negotiation.cc:299] Negotiation successful
0620 19:05:00.499672 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":93,"threads_started":1}
I20260620 19:05:00.500507 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.493973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.1.154.131:36395)
0620 19:05:00.494362 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:00.494372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:00.494391 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:05:00.494453 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:00.494458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:00.494613 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:00.495040 (+ 427us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:00.495150 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.496214 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.497950 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.498544 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.499370 (+ 826us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.499507 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:00.499552 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 19:05:00.500399 (+ 847us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":101,"threads_started":1}
I20260620 19:05:00.502844 3034 tablet_service.cc:1467] Processing CreateTablet for tablet afbb644f63d340b5aab29da5d5631f22 (DEFAULT_TABLE table=test-table [id=f2fe9f9a913048739200b13c13e37b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:00.503115 3174 tablet_service.cc:1467] Processing CreateTablet for tablet afbb644f63d340b5aab29da5d5631f22 (DEFAULT_TABLE table=test-table [id=f2fe9f9a913048739200b13c13e37b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:00.503105 2894 tablet_service.cc:1467] Processing CreateTablet for tablet afbb644f63d340b5aab29da5d5631f22 (DEFAULT_TABLE table=test-table [id=f2fe9f9a913048739200b13c13e37b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:00.503846 3034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afbb644f63d340b5aab29da5d5631f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:00.504046 3174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afbb644f63d340b5aab29da5d5631f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:00.504040 2894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afbb644f63d340b5aab29da5d5631f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:00.510569 3263 tablet_bootstrap.cc:492] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3: Bootstrap starting.
I20260620 19:05:00.511634 3264 tablet_bootstrap.cc:492] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2: Bootstrap starting.
I20260620 19:05:00.512910 3263 tablet_bootstrap.cc:654] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:00.513305 3264 tablet_bootstrap.cc:654] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:00.513696 3263 log.cc:824] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:00.513909 3265 tablet_bootstrap.cc:492] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0: Bootstrap starting.
I20260620 19:05:00.514086 3264 log.cc:824] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:00.515993 3263 tablet_bootstrap.cc:492] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3: No bootstrap required, opened a new log
I20260620 19:05:00.516033 3265 tablet_bootstrap.cc:654] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:00.516209 3263 ts_tablet_manager.cc:1397] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:05:00.516799 3265 log.cc:824] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:00.517007 3264 tablet_bootstrap.cc:492] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2: No bootstrap required, opened a new log
I20260620 19:05:00.517190 3264 ts_tablet_manager.cc:1397] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 19:05:00.518659 3265 tablet_bootstrap.cc:492] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0: No bootstrap required, opened a new log
I20260620 19:05:00.518878 3265 ts_tablet_manager.cc:1397] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 19:05:00.520151 3263 raft_consensus.cc:348] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } }
I20260620 19:05:00.520411 3263 raft_consensus.cc:374] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:00.520470 3263 raft_consensus.cc:729] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395b12ffc77546468d7977123d6a58a3, State: Initialized, Role: FOLLOWER
I20260620 19:05:00.520442 3264 raft_consensus.cc:348] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } }
I20260620 19:05:00.520653 3264 raft_consensus.cc:374] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:00.520704 3264 raft_consensus.cc:729] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff042333a51246748ca1eb7bf3273cf2, State: Initialized, Role: FOLLOWER
I20260620 19:05:00.521037 3263 consensus_queue.cc:260] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } }
I20260620 19:05:00.521119 3264 consensus_queue.cc:260] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } }
I20260620 19:05:00.522058 3263 ts_tablet_manager.cc:1428] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:05:00.522421 3100 heartbeater.cc:503] Master 127.1.154.190:44331 was elected leader, sending a full tablet report...
I20260620 19:05:00.522707 2960 heartbeater.cc:503] Master 127.1.154.190:44331 was elected leader, sending a full tablet report...
I20260620 19:05:00.522840 3265 raft_consensus.cc:348] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } }
I20260620 19:05:00.523046 3265 raft_consensus.cc:374] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:00.523110 3265 raft_consensus.cc:729] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43492d48a9944626b60b2cbc95cfc5d0, State: Initialized, Role: FOLLOWER
I20260620 19:05:00.523474 3265 consensus_queue.cc:260] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } }
I20260620 19:05:00.523758 3264 ts_tablet_manager.cc:1428] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:05:00.524119 3240 heartbeater.cc:503] Master 127.1.154.190:44331 was elected leader, sending a full tablet report...
I20260620 19:05:00.525276 3265 ts_tablet_manager.cc:1428] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260620 19:05:00.538743 3271 raft_consensus.cc:482] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:00.538905 3271 raft_consensus.cc:504] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } }
I20260620 19:05:00.539979 3271 leader_election.cc:288] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 395b12ffc77546468d7977123d6a58a3 (127.1.154.130:39525), ff042333a51246748ca1eb7bf3273cf2 (127.1.154.129:38143)
I20260620 19:05:00.545289 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.540508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:38143 (local address 127.1.154.131:35835)
0620 19:05:00.540674 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:00.540692 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:00.540833 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:00.541134 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:00.541137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:00.541154 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:00.541306 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.541312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.542557 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.542561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.543838 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.543848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.544871 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.544874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.544960 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.544982 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:05:00.545056 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:05:00.545121 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 19:05:00.545325 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.540583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:35835 (local address 127.1.154.129:38143)
0620 19:05:00.540726 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:00.540733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:00.540888 (+ 155us) server_negotiation.cc:407] Connection header received
0620 19:05:00.540976 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:00.540979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:00.541022 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:00.541112 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:00.541426 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.542421 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.544033 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.544721 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.544871 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.544925 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:00.545091 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 19:05:00.545176 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 19:05:00.546124 2914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbb644f63d340b5aab29da5d5631f22" candidate_uuid: "43492d48a9944626b60b2cbc95cfc5d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff042333a51246748ca1eb7bf3273cf2" is_pre_election: true
I20260620 19:05:00.546463 2914 raft_consensus.cc:2454] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43492d48a9944626b60b2cbc95cfc5d0 in term 0.
I20260620 19:05:00.546487 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.540512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:39525 (local address 127.1.154.131:42761)
0620 19:05:00.541452 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:00.541478 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:00.541737 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:00.542148 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:00.542152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:00.542174 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:00.542350 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.542356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.543594 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.543597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.545167 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.545176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.546077 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.546081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.546188 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.546204 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 19:05:00.546294 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 19:05:00.546355 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":177,"thread_start_us":342,"threads_started":1}
I20260620 19:05:00.546478 3261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.540643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:42761 (local address 127.1.154.130:39525)
0620 19:05:00.540824 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:00.540830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:00.541627 (+ 797us) server_negotiation.cc:407] Connection header received
0620 19:05:00.541912 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:00.541916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:00.542005 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:00.542095 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:00.542527 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.543422 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.545315 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.545921 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.546061 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.546112 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:00.546308 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 19:05:00.546376 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 19:05:00.546957 3125 leader_election.cc:302] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43492d48a9944626b60b2cbc95cfc5d0, ff042333a51246748ca1eb7bf3273cf2; no voters:
I20260620 19:05:00.547092 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbb644f63d340b5aab29da5d5631f22" candidate_uuid: "43492d48a9944626b60b2cbc95cfc5d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395b12ffc77546468d7977123d6a58a3" is_pre_election: true
I20260620 19:05:00.547254 3271 raft_consensus.cc:2781] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:00.547362 3271 raft_consensus.cc:482] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:00.547336 3051 raft_consensus.cc:2454] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43492d48a9944626b60b2cbc95cfc5d0 in term 0.
I20260620 19:05:00.547423 3271 raft_consensus.cc:3037] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:00.548525 3271 raft_consensus.cc:504] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } }
I20260620 19:05:00.548869 3271 leader_election.cc:288] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [CANDIDATE]: Term 1 election: Requested vote from peers 395b12ffc77546468d7977123d6a58a3 (127.1.154.130:39525), ff042333a51246748ca1eb7bf3273cf2 (127.1.154.129:38143)
I20260620 19:05:00.549393 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbb644f63d340b5aab29da5d5631f22" candidate_uuid: "43492d48a9944626b60b2cbc95cfc5d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395b12ffc77546468d7977123d6a58a3"
I20260620 19:05:00.549402 2914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbb644f63d340b5aab29da5d5631f22" candidate_uuid: "43492d48a9944626b60b2cbc95cfc5d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff042333a51246748ca1eb7bf3273cf2"
I20260620 19:05:00.549522 2914 raft_consensus.cc:3037] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:00.549522 3051 raft_consensus.cc:3037] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:00.550861 2914 raft_consensus.cc:2454] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43492d48a9944626b60b2cbc95cfc5d0 in term 1.
I20260620 19:05:00.551199 3125 leader_election.cc:302] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43492d48a9944626b60b2cbc95cfc5d0, ff042333a51246748ca1eb7bf3273cf2; no voters:
I20260620 19:05:00.551229 3051 raft_consensus.cc:2454] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43492d48a9944626b60b2cbc95cfc5d0 in term 1.
I20260620 19:05:00.551411 3271 raft_consensus.cc:2781] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:00.551651 3271 raft_consensus.cc:686] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [term 1 LEADER]: Becoming Leader. State: Replica: 43492d48a9944626b60b2cbc95cfc5d0, State: Running, Role: LEADER
I20260620 19:05:00.551923 3271 consensus_queue.cc:237] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } }
I20260620 19:05:00.555065 2756 catalog_manager.cc:5495] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43492d48a9944626b60b2cbc95cfc5d0 (127.1.154.131). New cstate: current_term: 1 leader_uuid: "43492d48a9944626b60b2cbc95cfc5d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } health_report { overall_health: UNKNOWN } } }
I20260620 19:05:00.586633 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.582411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37342 (local address 127.1.154.131:36395)
0620 19:05:00.582612 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:00.582616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:00.582629 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:05:00.582690 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:00.582692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:00.582752 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:00.582859 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:00.583264 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.583836 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.584839 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.585118 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.585251 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.585675 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:00.586109 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:00.586260 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:00.586394 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 19:05:00.586456 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 19:05:00.593720 2914 raft_consensus.cc:1261] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2 [term 1 FOLLOWER]: Refusing update from remote peer 43492d48a9944626b60b2cbc95cfc5d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:00.593720 3051 raft_consensus.cc:1261] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3 [term 1 FOLLOWER]: Refusing update from remote peer 43492d48a9944626b60b2cbc95cfc5d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:00.594599 3273 consensus_queue.cc:1035] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:00.594841 3271 consensus_queue.cc:1035] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:00.615931 2756 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:00.616129 2756 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:00.618767 2756 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4: Sending DeleteTablet for 3 replicas of tablet afbb644f63d340b5aab29da5d5631f22
I20260620 19:05:00.619532 3174 tablet_service.cc:1514] Processing DeleteTablet for tablet afbb644f63d340b5aab29da5d5631f22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:00 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:37338
I20260620 19:05:00.619536 3034 tablet_service.cc:1514] Processing DeleteTablet for tablet afbb644f63d340b5aab29da5d5631f22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:00 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:36134
I20260620 19:05:00.619762 2894 tablet_service.cc:1514] Processing DeleteTablet for tablet afbb644f63d340b5aab29da5d5631f22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:00 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:37392
I20260620 19:05:00.620067 3286 tablet_replica.cc:330] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3: stopping tablet replica
I20260620 19:05:00.620398 3286 raft_consensus.cc:2229] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:00.620590 2756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:05:00.620726 3286 raft_consensus.cc:2258] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 19:05:00.621049 2756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:00.621806 3288 tablet_replica.cc:330] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2: stopping tablet replica
I20260620 19:05:00.622047 3287 tablet_replica.cc:330] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0: stopping tablet replica
I20260620 19:05:00.622370 3287 raft_consensus.cc:2229] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:05:00.622402 3286 ts_tablet_manager.cc:1905] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:00.622716 3288 raft_consensus.cc:2229] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:00.623142 3287 raft_consensus.cc:2258] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:00.623478 3288 raft_consensus.cc:2258] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:00.625366 3287 ts_tablet_manager.cc:1905] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:00.625793 3286 ts_tablet_manager.cc:1918] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:00.625921 3286 log.cc:1196] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/afbb644f63d340b5aab29da5d5631f22
I20260620 19:05:00.626313 3286 ts_tablet_manager.cc:1939] T afbb644f63d340b5aab29da5d5631f22 P 395b12ffc77546468d7977123d6a58a3: Deleting consensus metadata
I20260620 19:05:00.626534 3288 ts_tablet_manager.cc:1905] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:00.627012 3034 tablet_service.cc:1467] Processing CreateTablet for tablet 62cfce843203441e80d94b57ce691741 (DEFAULT_TABLE table=test-table [id=973cffd5050846baa190017174d8eb58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:00.627259 3034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62cfce843203441e80d94b57ce691741. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:00.627851 2743 catalog_manager.cc:4841] TS 395b12ffc77546468d7977123d6a58a3 (127.1.154.130:39525): tablet afbb644f63d340b5aab29da5d5631f22 (table test-table [id=f2fe9f9a913048739200b13c13e37b24]) successfully deleted
I20260620 19:05:00.629415 3263 tablet_bootstrap.cc:492] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3: Bootstrap starting.
I20260620 19:05:00.627092 3174 tablet_service.cc:1467] Processing CreateTablet for tablet 62cfce843203441e80d94b57ce691741 (DEFAULT_TABLE table=test-table [id=973cffd5050846baa190017174d8eb58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:00.627249 2894 tablet_service.cc:1467] Processing CreateTablet for tablet 62cfce843203441e80d94b57ce691741 (DEFAULT_TABLE table=test-table [id=973cffd5050846baa190017174d8eb58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:00.630055 2894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62cfce843203441e80d94b57ce691741. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:00.630488 3263 tablet_bootstrap.cc:654] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:00.632472 3263 tablet_bootstrap.cc:492] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3: No bootstrap required, opened a new log
I20260620 19:05:00.632565 3263 ts_tablet_manager.cc:1397] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:00.632666 3264 tablet_bootstrap.cc:492] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2: Bootstrap starting.
I20260620 19:05:00.633625 3264 tablet_bootstrap.cc:654] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:00.633491 3263 raft_consensus.cc:348] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:00.633688 3263 raft_consensus.cc:374] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:00.633735 3263 raft_consensus.cc:729] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395b12ffc77546468d7977123d6a58a3, State: Initialized, Role: FOLLOWER
I20260620 19:05:00.634073 3263 consensus_queue.cc:260] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:00.632469 3287 ts_tablet_manager.cc:1918] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:00.632931 3174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62cfce843203441e80d94b57ce691741. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:00.634404 3263 ts_tablet_manager.cc:1428] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:05:00.634404 3287 log.cc:1196] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/afbb644f63d340b5aab29da5d5631f22
I20260620 19:05:00.634809 3287 ts_tablet_manager.cc:1939] T afbb644f63d340b5aab29da5d5631f22 P 43492d48a9944626b60b2cbc95cfc5d0: Deleting consensus metadata
I20260620 19:05:00.636042 2744 catalog_manager.cc:4841] TS 43492d48a9944626b60b2cbc95cfc5d0 (127.1.154.131:36395): tablet afbb644f63d340b5aab29da5d5631f22 (table test-table [id=f2fe9f9a913048739200b13c13e37b24]) successfully deleted
I20260620 19:05:00.636612 3265 tablet_bootstrap.cc:492] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0: Bootstrap starting.
I20260620 19:05:00.636861 3264 tablet_bootstrap.cc:492] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2: No bootstrap required, opened a new log
I20260620 19:05:00.636940 3264 ts_tablet_manager.cc:1397] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260620 19:05:00.637393 3264 raft_consensus.cc:348] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:00.637516 3264 raft_consensus.cc:374] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:00.637547 3264 raft_consensus.cc:729] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff042333a51246748ca1eb7bf3273cf2, State: Initialized, Role: FOLLOWER
I20260620 19:05:00.637576 3265 tablet_bootstrap.cc:654] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:00.637684 3264 consensus_queue.cc:260] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:00.637946 3264 ts_tablet_manager.cc:1428] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:05:00.638757 3265 tablet_bootstrap.cc:492] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0: No bootstrap required, opened a new log
I20260620 19:05:00.638860 3265 ts_tablet_manager.cc:1397] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:05:00.639302 3265 raft_consensus.cc:348] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:00.639412 3265 raft_consensus.cc:374] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:00.639452 3265 raft_consensus.cc:729] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43492d48a9944626b60b2cbc95cfc5d0, State: Initialized, Role: FOLLOWER
I20260620 19:05:00.639582 3265 consensus_queue.cc:260] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:00.639860 3265 ts_tablet_manager.cc:1428] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:05:00.642923 3288 ts_tablet_manager.cc:1918] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:00.643009 3288 log.cc:1196] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-0/wal/wals/afbb644f63d340b5aab29da5d5631f22
I20260620 19:05:00.643296 3288 ts_tablet_manager.cc:1939] T afbb644f63d340b5aab29da5d5631f22 P ff042333a51246748ca1eb7bf3273cf2: Deleting consensus metadata
I20260620 19:05:00.644456 2741 catalog_manager.cc:4841] TS ff042333a51246748ca1eb7bf3273cf2 (127.1.154.129:38143): tablet afbb644f63d340b5aab29da5d5631f22 (table test-table [id=f2fe9f9a913048739200b13c13e37b24]) successfully deleted
W20260620 19:05:00.686081 3241 tablet.cc:2369] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:05:00.698354 3101 tablet.cc:2369] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:05:00.699414 2961 tablet.cc:2369] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:05:00.909320 3269 raft_consensus.cc:482] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:00.909528 3269 raft_consensus.cc:504] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:00.910684 3269 leader_election.cc:288] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff042333a51246748ca1eb7bf3273cf2 (127.1.154.129:38143), 43492d48a9944626b60b2cbc95cfc5d0 (127.1.154.131:36395)
I20260620 19:05:00.914583 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.911212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:56217 (local address 127.1.154.129:38143)
0620 19:05:00.911326 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:00.911332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:00.911429 (+ 97us) server_negotiation.cc:407] Connection header received
0620 19:05:00.911622 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:00.911626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:00.911677 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:00.911758 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:00.912006 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.912587 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.913636 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.914040 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.914168 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.914218 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:00.914392 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 19:05:00.914455 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:05:00.914625 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.911093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:48801 (local address 127.1.154.131:36395)
0620 19:05:00.911236 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:00.911242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:00.911629 (+ 387us) server_negotiation.cc:407] Connection header received
0620 19:05:00.911779 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:00.911781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:00.911830 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:00.911913 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:00.912153 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.912704 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.913714 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.914089 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.914215 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.914264 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:00.914443 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 19:05:00.914502 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:05:00.914654 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.911001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:36395 (local address 127.1.154.130:48801)
0620 19:05:00.911531 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:00.911543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:00.911662 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:00.911921 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:00.911923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:00.911938 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:00.912047 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.912052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.912815 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.912820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.913563 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.913570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.914210 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.914214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.914311 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.914338 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:05:00.914427 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 19:05:00.914492 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":252,"thread_start_us":76,"threads_started":1}
I20260620 19:05:00.914963 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.911001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:38143 (local address 127.1.154.130:56217)
0620 19:05:00.911301 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:00.911364 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:00.911493 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:00.911768 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:00.911771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:00.911786 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:00.911906 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.911911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.912687 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.912692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.913477 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.913485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.914163 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.914167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.914260 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.914281 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:05:00.914378 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 19:05:00.914448 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":98,"threads_started":1}
I20260620 19:05:00.915192 3194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62cfce843203441e80d94b57ce691741" candidate_uuid: "395b12ffc77546468d7977123d6a58a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43492d48a9944626b60b2cbc95cfc5d0" is_pre_election: true
I20260620 19:05:00.915416 3194 raft_consensus.cc:2454] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395b12ffc77546468d7977123d6a58a3 in term 0.
I20260620 19:05:00.915432 2914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62cfce843203441e80d94b57ce691741" candidate_uuid: "395b12ffc77546468d7977123d6a58a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff042333a51246748ca1eb7bf3273cf2" is_pre_election: true
I20260620 19:05:00.915584 2914 raft_consensus.cc:2454] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395b12ffc77546468d7977123d6a58a3 in term 0.
I20260620 19:05:00.915786 2988 leader_election.cc:302] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 395b12ffc77546468d7977123d6a58a3, 43492d48a9944626b60b2cbc95cfc5d0; no voters:
I20260620 19:05:00.916056 3269 raft_consensus.cc:2781] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:00.916115 3269 raft_consensus.cc:482] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:00.916152 3269 raft_consensus.cc:3037] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:00.917098 3269 raft_consensus.cc:504] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:00.917456 3269 leader_election.cc:288] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [CANDIDATE]: Term 1 election: Requested vote from peers ff042333a51246748ca1eb7bf3273cf2 (127.1.154.129:38143), 43492d48a9944626b60b2cbc95cfc5d0 (127.1.154.131:36395)
I20260620 19:05:00.917881 2914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62cfce843203441e80d94b57ce691741" candidate_uuid: "395b12ffc77546468d7977123d6a58a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff042333a51246748ca1eb7bf3273cf2"
I20260620 19:05:00.917963 2914 raft_consensus.cc:3037] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:00.918190 3194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62cfce843203441e80d94b57ce691741" candidate_uuid: "395b12ffc77546468d7977123d6a58a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43492d48a9944626b60b2cbc95cfc5d0"
I20260620 19:05:00.918308 3194 raft_consensus.cc:3037] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:00.919031 2914 raft_consensus.cc:2454] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 395b12ffc77546468d7977123d6a58a3 in term 1.
I20260620 19:05:00.919355 2985 leader_election.cc:302] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 395b12ffc77546468d7977123d6a58a3, ff042333a51246748ca1eb7bf3273cf2; no voters:
I20260620 19:05:00.919499 3194 raft_consensus.cc:2454] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 395b12ffc77546468d7977123d6a58a3 in term 1.
I20260620 19:05:00.919544 3269 raft_consensus.cc:2781] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:00.919747 3269 raft_consensus.cc:686] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [term 1 LEADER]: Becoming Leader. State: Replica: 395b12ffc77546468d7977123d6a58a3, State: Running, Role: LEADER
I20260620 19:05:00.919981 3269 consensus_queue.cc:237] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:00.922717 2755 catalog_manager.cc:5495] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 395b12ffc77546468d7977123d6a58a3 (127.1.154.130). New cstate: current_term: 1 leader_uuid: "395b12ffc77546468d7977123d6a58a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } health_report { overall_health: UNKNOWN } } }
Jun 20 19:05:00 dist-test-slave-6wrf krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982299, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:00.964483 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.955380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.1.154.190:44331)
0620 19:05:00.955539 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:00.955544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:00.955775 (+ 231us) server_negotiation.cc:407] Connection header received
0620 19:05:00.955894 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:00.955897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:00.955942 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:00.956021 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:00.956580 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.957389 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.958316 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.958534 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.958659 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.961141 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:00.961145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:00.961149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:00.961175 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:00.962708 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:00.963186 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:00.963192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:00.963197 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:00.963276 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:00.963547 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:00.963552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:00.963553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:00.963789 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:00.963898 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:00.964180 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 19:05:00.964344 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":66}
I20260620 19:05:00.964483 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:00.955380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:44331 (local address 127.0.0.1:50772)
0620 19:05:00.955676 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:00.955691 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:00.955797 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:00.956047 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:00.956049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:00.956387 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:00.956472 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.956477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.957514 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.957517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.958181 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:00.958190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:00.958662 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:00.958666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:00.958744 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:00.959352 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:00.959364 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:00.960994 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:00.962843 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:00.962853 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:00.962860 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:00.963086 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:00.963371 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:00.963374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:00.963376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:00.963460 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:00.963897 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:00.963906 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:00.964015 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 19:05:00.964136 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 19:05:00.964288 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":136,"threads_started":1}
I20260620 19:05:00.970010 2755 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:50772:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:05:00.971194 2755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:00.976473 3034 tablet_service.cc:1467] Processing CreateTablet for tablet 5974748cf4ea44549b0b6cb6e6e76318 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e312b4743ded4728befdbd1c50995d3f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:05:00.976473 2894 tablet_service.cc:1467] Processing CreateTablet for tablet 5974748cf4ea44549b0b6cb6e6e76318 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e312b4743ded4728befdbd1c50995d3f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:05:00.976600 3174 tablet_service.cc:1467] Processing CreateTablet for tablet 5974748cf4ea44549b0b6cb6e6e76318 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e312b4743ded4728befdbd1c50995d3f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:05:00.976858 2894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5974748cf4ea44549b0b6cb6e6e76318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:00.976861 3034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5974748cf4ea44549b0b6cb6e6e76318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:00.976936 3174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5974748cf4ea44549b0b6cb6e6e76318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:00.979473 3263 tablet_bootstrap.cc:492] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3: Bootstrap starting.
I20260620 19:05:00.979503 3265 tablet_bootstrap.cc:492] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0: Bootstrap starting.
I20260620 19:05:00.980446 3265 tablet_bootstrap.cc:654] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:00.980502 3263 tablet_bootstrap.cc:654] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:00.981071 3264 tablet_bootstrap.cc:492] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2: Bootstrap starting.
I20260620 19:05:00.981837 3265 tablet_bootstrap.cc:492] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0: No bootstrap required, opened a new log
I20260620 19:05:00.981940 3265 ts_tablet_manager.cc:1397] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:05:00.982012 3264 tablet_bootstrap.cc:654] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:00.982174 3263 tablet_bootstrap.cc:492] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3: No bootstrap required, opened a new log
I20260620 19:05:00.982257 3263 ts_tablet_manager.cc:1397] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:00.982374 3265 raft_consensus.cc:348] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:00.982499 3265 raft_consensus.cc:374] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:00.982537 3265 raft_consensus.cc:729] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43492d48a9944626b60b2cbc95cfc5d0, State: Initialized, Role: FOLLOWER
I20260620 19:05:00.982671 3265 consensus_queue.cc:260] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:00.982928 3265 ts_tablet_manager.cc:1428] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:05:00.982867 3263 raft_consensus.cc:348] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:00.982999 3263 raft_consensus.cc:374] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:00.983042 3263 raft_consensus.cc:729] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395b12ffc77546468d7977123d6a58a3, State: Initialized, Role: FOLLOWER
I20260620 19:05:00.983371 3263 consensus_queue.cc:260] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:00.983484 3271 tablet_replica.cc:439] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } }
I20260620 19:05:00.983616 3271 tablet_replica.cc:442] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:00.983593 3269 tablet_replica.cc:439] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } }
I20260620 19:05:00.983693 3269 tablet_replica.cc:442] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:00.983914 3263 ts_tablet_manager.cc:1428] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:05:00.984812 3264 tablet_bootstrap.cc:492] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2: No bootstrap required, opened a new log
I20260620 19:05:00.984875 3264 ts_tablet_manager.cc:1397] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 19:05:00.985308 3264 raft_consensus.cc:348] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:00.985417 3264 raft_consensus.cc:374] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:00.985453 3264 raft_consensus.cc:729] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff042333a51246748ca1eb7bf3273cf2, State: Initialized, Role: FOLLOWER
I20260620 19:05:00.985566 3264 consensus_queue.cc:260] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:00.985911 3264 ts_tablet_manager.cc:1428] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:05:00.986101 3270 tablet_replica.cc:439] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } }
I20260620 19:05:00.986248 3270 tablet_replica.cc:442] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:01.092262 3271 raft_consensus.cc:482] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:01.092401 3271 raft_consensus.cc:504] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:01.092890 3271 leader_election.cc:288] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff042333a51246748ca1eb7bf3273cf2 (127.1.154.129:38143), 395b12ffc77546468d7977123d6a58a3 (127.1.154.130:39525)
I20260620 19:05:01.093336 2914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5974748cf4ea44549b0b6cb6e6e76318" candidate_uuid: "43492d48a9944626b60b2cbc95cfc5d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff042333a51246748ca1eb7bf3273cf2" is_pre_election: true
I20260620 19:05:01.093420 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5974748cf4ea44549b0b6cb6e6e76318" candidate_uuid: "43492d48a9944626b60b2cbc95cfc5d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395b12ffc77546468d7977123d6a58a3" is_pre_election: true
I20260620 19:05:01.093526 2914 raft_consensus.cc:2454] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43492d48a9944626b60b2cbc95cfc5d0 in term 0.
I20260620 19:05:01.093570 3051 raft_consensus.cc:2454] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43492d48a9944626b60b2cbc95cfc5d0 in term 0.
I20260620 19:05:01.093940 3125 leader_election.cc:302] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 395b12ffc77546468d7977123d6a58a3, 43492d48a9944626b60b2cbc95cfc5d0, ff042333a51246748ca1eb7bf3273cf2; no voters:
I20260620 19:05:01.094168 3271 raft_consensus.cc:2781] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:01.094262 3271 raft_consensus.cc:482] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:01.094321 3271 raft_consensus.cc:3037] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:01.095348 3271 raft_consensus.cc:504] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:01.095737 3271 leader_election.cc:288] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [CANDIDATE]: Term 1 election: Requested vote from peers ff042333a51246748ca1eb7bf3273cf2 (127.1.154.129:38143), 395b12ffc77546468d7977123d6a58a3 (127.1.154.130:39525)
I20260620 19:05:01.096442 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5974748cf4ea44549b0b6cb6e6e76318" candidate_uuid: "43492d48a9944626b60b2cbc95cfc5d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395b12ffc77546468d7977123d6a58a3"
I20260620 19:05:01.096442 2914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5974748cf4ea44549b0b6cb6e6e76318" candidate_uuid: "43492d48a9944626b60b2cbc95cfc5d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff042333a51246748ca1eb7bf3273cf2"
I20260620 19:05:01.096655 2914 raft_consensus.cc:3037] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:01.096655 3051 raft_consensus.cc:3037] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:01.098351 3051 raft_consensus.cc:2454] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43492d48a9944626b60b2cbc95cfc5d0 in term 1.
I20260620 19:05:01.098351 2914 raft_consensus.cc:2454] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43492d48a9944626b60b2cbc95cfc5d0 in term 1.
I20260620 19:05:01.098917 3125 leader_election.cc:302] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 395b12ffc77546468d7977123d6a58a3, 43492d48a9944626b60b2cbc95cfc5d0, ff042333a51246748ca1eb7bf3273cf2; no voters:
I20260620 19:05:01.099210 3271 raft_consensus.cc:2781] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:01.099320 3271 raft_consensus.cc:686] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [term 1 LEADER]: Becoming Leader. State: Replica: 43492d48a9944626b60b2cbc95cfc5d0, State: Running, Role: LEADER
I20260620 19:05:01.099454 3271 consensus_queue.cc:237] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } }
I20260620 19:05:01.100144 3273 tablet_replica.cc:439] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0: TxnStatusTablet state changed. Reason: New leader 43492d48a9944626b60b2cbc95cfc5d0. Latest consensus state: current_term: 1 leader_uuid: "43492d48a9944626b60b2cbc95cfc5d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } }
I20260620 19:05:01.100270 3273 tablet_replica.cc:442] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:05:01.100546 3306 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:05:01.100708 3306 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:05:01.101470 2755 catalog_manager.cc:5495] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43492d48a9944626b60b2cbc95cfc5d0 (127.1.154.131). New cstate: current_term: 1 leader_uuid: "43492d48a9944626b60b2cbc95cfc5d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } health_report { overall_health: HEALTHY } } }
I20260620 19:05:01.115397 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:01.110633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.1.154.131:36395)
0620 19:05:01.110766 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:01.110770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:01.110852 (+ 82us) server_negotiation.cc:407] Connection header received
0620 19:05:01.111092 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:01.111096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:01.111188 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:01.111273 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:01.111689 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.112781 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.114144 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.114477 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.114565 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:01.114936 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:01.115017 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:01.115074 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:01.115232 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 19:05:01.115294 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:05:01.115420 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:01.110548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:36395 (local address 127.0.0.1:37346)
0620 19:05:01.110792 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:01.110807 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:01.110917 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:01.111390 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:01.111393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:01.111401 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:05:01.111499 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:01.111505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.112952 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.112956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:01.114007 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:01.114018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.114652 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.114656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:01.114738 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:01.114790 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:05:01.115110 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:05:01.115124 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 19:05:01.115194 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:05:01.115273 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260620 19:05:01.118256 3051 raft_consensus.cc:1261] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3 [term 1 FOLLOWER]: Refusing update from remote peer 43492d48a9944626b60b2cbc95cfc5d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:01.118427 2914 raft_consensus.cc:1261] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2 [term 1 FOLLOWER]: Refusing update from remote peer 43492d48a9944626b60b2cbc95cfc5d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:01.118636 3271 consensus_queue.cc:1035] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:01.118825 3273 consensus_queue.cc:1035] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:01.119918 3269 tablet_replica.cc:439] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3: TxnStatusTablet state changed. Reason: New leader 43492d48a9944626b60b2cbc95cfc5d0. Latest consensus state: current_term: 1 leader_uuid: "43492d48a9944626b60b2cbc95cfc5d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } }
I20260620 19:05:01.120035 3269 tablet_replica.cc:442] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:01.120087 3270 tablet_replica.cc:439] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2: TxnStatusTablet state changed. Reason: New leader 43492d48a9944626b60b2cbc95cfc5d0. Latest consensus state: current_term: 1 leader_uuid: "43492d48a9944626b60b2cbc95cfc5d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } }
I20260620 19:05:01.120179 3270 tablet_replica.cc:442] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:01.122395 3273 tablet_replica.cc:439] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43492d48a9944626b60b2cbc95cfc5d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } }
I20260620 19:05:01.122499 3273 tablet_replica.cc:442] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:05:01.122573 3309 mvcc.cc:205] Tried to move back new op lower bound from 7298999505376223232 to 7298999505304899584. Current Snapshot: MvccSnapshot[applied={T|T < 7298999505376223232}]
I20260620 19:05:01.122924 3270 tablet_replica.cc:439] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43492d48a9944626b60b2cbc95cfc5d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } }
I20260620 19:05:01.123029 3270 tablet_replica.cc:442] T 5974748cf4ea44549b0b6cb6e6e76318 P ff042333a51246748ca1eb7bf3273cf2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:01.123041 3271 tablet_replica.cc:439] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43492d48a9944626b60b2cbc95cfc5d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } }
I20260620 19:05:01.123128 3271 tablet_replica.cc:442] T 5974748cf4ea44549b0b6cb6e6e76318 P 43492d48a9944626b60b2cbc95cfc5d0: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:05:01.124338 3269 tablet_replica.cc:439] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43492d48a9944626b60b2cbc95cfc5d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 } } peers { permanent_uuid: "395b12ffc77546468d7977123d6a58a3" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39525 } } peers { permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 } } }
I20260620 19:05:01.124447 3269 tablet_replica.cc:442] T 5974748cf4ea44549b0b6cb6e6e76318 P 395b12ffc77546468d7977123d6a58a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:01.124782 3311 mvcc.cc:205] Tried to move back new op lower bound from 7298999505376223232 to 7298999505304899584. Current Snapshot: MvccSnapshot[applied={T|T < 7298999505376223232 or (T in {7298999505376223232})}]
I20260620 19:05:01.125010 3310 mvcc.cc:205] Tried to move back new op lower bound from 7298999505376223232 to 7298999505304899584. Current Snapshot: MvccSnapshot[applied={T|T < 7298999505376223232 or (T in {7298999505376223232})}]
I20260620 19:05:01.134198 3319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:01.130459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36148 (local address 127.1.154.130:39525)
0620 19:05:01.130668 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:01.130672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:01.130741 (+ 69us) server_negotiation.cc:407] Connection header received
0620 19:05:01.130952 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:01.130955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:01.131019 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:01.131138 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:01.131388 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.131893 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.132717 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.132955 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.133086 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:01.133314 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:01.133807 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:01.133926 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:01.133993 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 19:05:01.134050 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260620 19:05:01.145371 3322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:01.141544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:36395 (local address 127.1.154.130:43707)
0620 19:05:01.142136 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:01.142155 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:01.142302 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:01.142625 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:01.142628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:01.142635 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:05:01.142777 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:01.142787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.143478 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.143484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:01.144100 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:01.144108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.144636 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.144649 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:01.144737 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:01.144777 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:05:01.145087 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:05:01.145095 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:05:01.145156 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:05:01.145230 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":224,"threads_started":1}
I20260620 19:05:01.145351 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:01.141691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:43707 (local address 127.1.154.131:36395)
0620 19:05:01.141801 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:01.141825 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:01.142247 (+ 422us) server_negotiation.cc:407] Connection header received
0620 19:05:01.142416 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:01.142419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:01.142516 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:01.142609 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:01.142871 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.143372 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.144218 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.144510 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.144603 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:01.144912 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:01.144995 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:01.145074 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:01.145166 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 19:05:01.145212 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 19:05:01.151723 2914 raft_consensus.cc:1261] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2 [term 1 FOLLOWER]: Refusing update from remote peer 395b12ffc77546468d7977123d6a58a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:01.151832 3194 raft_consensus.cc:1261] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0 [term 1 FOLLOWER]: Refusing update from remote peer 395b12ffc77546468d7977123d6a58a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:01.152233 3294 consensus_queue.cc:1035] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff042333a51246748ca1eb7bf3273cf2" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:01.152436 3269 consensus_queue.cc:1035] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43492d48a9944626b60b2cbc95cfc5d0" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:01.176724 3319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:01.172657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:59485 (local address 127.1.154.130:39525)
0620 19:05:01.172808 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:01.172813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:01.172921 (+ 108us) server_negotiation.cc:407] Connection header received
0620 19:05:01.173126 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:01.173129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:01.173177 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:01.173253 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:01.173623 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.174403 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.175623 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.175835 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.175933 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:01.176239 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:01.176354 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:01.176426 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:01.176528 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 19:05:01.176585 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 19:05:01.176740 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:01.172350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:39525 (local address 127.1.154.131:59485)
0620 19:05:01.172815 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:01.172833 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:01.172993 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:01.173368 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:01.173372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:01.173381 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:05:01.173483 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:01.173489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.174540 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.174543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:01.175500 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:01.175510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.175963 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.175967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:01.176051 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:01.176114 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:05:01.176445 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:05:01.176450 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:05:01.176514 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:05:01.176592 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":79,"threads_started":1}
I20260620 19:05:01.205200 2757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:01.205384 2757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:01.207943 2757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a0aeaf2c1f04b86aaba045c477edde4: Sending DeleteTablet for 3 replicas of tablet 62cfce843203441e80d94b57ce691741
I20260620 19:05:01.208580 3034 tablet_service.cc:1514] Processing DeleteTablet for tablet 62cfce843203441e80d94b57ce691741 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:01 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:36134
I20260620 19:05:01.208756 3174 tablet_service.cc:1514] Processing DeleteTablet for tablet 62cfce843203441e80d94b57ce691741 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:01 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:37338
I20260620 19:05:01.208751 2894 tablet_service.cc:1514] Processing DeleteTablet for tablet 62cfce843203441e80d94b57ce691741 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:01 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:37392
I20260620 19:05:01.215685 3331 tablet_replica.cc:330] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3: stopping tablet replica
I20260620 19:05:01.215910 3333 tablet_replica.cc:330] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0: stopping tablet replica
I20260620 19:05:01.216089 3333 raft_consensus.cc:2229] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:01.216265 3331 raft_consensus.cc:2229] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:05:01.216450 3333 raft_consensus.cc:2258] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:01.216816 3331 raft_consensus.cc:2258] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 19:05:01 dist-test-slave-6wrf krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982300, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.154.129@KRBTEST.COM
I20260620 19:05:01.218206 3333 ts_tablet_manager.cc:1905] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:01.218415 3331 ts_tablet_manager.cc:1905] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:01.223917 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:01.210360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.1.154.129:38143)
0620 19:05:01.210471 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:01.210475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:01.210966 (+ 491us) server_negotiation.cc:407] Connection header received
0620 19:05:01.211160 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:01.211164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:01.211215 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:01.211268 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:01.212299 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.213109 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.214349 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.214677 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.214753 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:01.218695 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:01.218708 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:01.218724 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:01.218752 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:01.221537 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:01.222253 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:01.222260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:01.222262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:01.222355 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:01.222732 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:01.222735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:01.222737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:01.223024 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:01.223210 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:01.223584 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 19:05:01.223726 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 19:05:01.224776 2953 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37402
I20260620 19:05:01.227774 3333 ts_tablet_manager.cc:1918] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:05:01.227860 3333 log.cc:1196] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/62cfce843203441e80d94b57ce691741
I20260620 19:05:01.228142 3333 ts_tablet_manager.cc:1939] T 62cfce843203441e80d94b57ce691741 P 43492d48a9944626b60b2cbc95cfc5d0: Deleting consensus metadata
I20260620 19:05:01.228899 2744 catalog_manager.cc:4841] TS 43492d48a9944626b60b2cbc95cfc5d0 (127.1.154.131:36395): tablet 62cfce843203441e80d94b57ce691741 (table test-table [id=973cffd5050846baa190017174d8eb58]) successfully deleted
I20260620 19:05:01.233292 3331 ts_tablet_manager.cc:1918] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:05:01.233395 3331 log.cc:1196] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/62cfce843203441e80d94b57ce691741
I20260620 19:05:01.233786 3332 tablet_replica.cc:330] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2: stopping tablet replica
I20260620 19:05:01.233937 3332 raft_consensus.cc:2229] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:01.234057 3332 raft_consensus.cc:2258] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:01.234545 3331 ts_tablet_manager.cc:1939] T 62cfce843203441e80d94b57ce691741 P 395b12ffc77546468d7977123d6a58a3: Deleting consensus metadata
I20260620 19:05:01.235709 3332 ts_tablet_manager.cc:1905] T 62cfce843203441e80d94b57ce691741 P ff042333a51246748ca1eb7bf3273cf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:01.237361 2743 catalog_manager.cc:4841] TS 395b12ffc77546468d7977123d6a58a3 (127.1.154.130:39525): tablet 62cfce843203441e80d94b57ce691741 (table test-table [id=973cffd5050846baa190017174d8eb58]) successfully deleted
I20260620 19:05:01.237694 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:01.227127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50786 (local address 127.1.154.190:44331)
0620 19:05:01.227244 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:01.227248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:01.227658 (+ 410us) server_negotiation.cc:407] Connection header received
0620 19:05:01.227827 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:01.227831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:01.227876 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:01.227925 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:01.229512 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.230346 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.231504 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:01.231846 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:01.231923 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:01.233834 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:01.233838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:01.233841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:01.233866 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:01.235556 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:01.236102 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:01.236106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:01.236108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:01.236188 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:01.236478 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:01.236481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:01.236483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:01.236708 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:01.236812 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:01.237170 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 19:05:01.237282 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 19:05:01.238147 2757 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50786
I20260620 19:05:01.239758 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 2829
W20260620 19:05:01.246104 2741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:38143 (error 108)
I20260620 19:05:01.246910 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 2969
I20260620 19:05:01.249055 3339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:01.248567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:38143 (local address 127.0.0.1:37406)
0620 19:05:01.248842 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:01.248923 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:38143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
W20260620 19:05:01.249490 2741 catalog_manager.cc:4584] TS ff042333a51246748ca1eb7bf3273cf2 (127.1.154.129:38143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62cfce843203441e80d94b57ce691741: Network error: Client connection negotiation failed: client connection to 127.1.154.129:38143: connect: Connection refused (error 111)
I20260620 19:05:01.254061 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 3109
I20260620 19:05:01.260358 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 2725
2026-06-20T19:05:01Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3027 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:05:01 dist-test-slave-6wrf krb5kdc[3342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:05:01 dist-test-slave-6wrf krb5kdc[3342](info): set up 2 sockets
Jun 20 19:05:01 dist-test-slave-6wrf krb5kdc[3342](info): commencing operation
krb5kdc: starting...
W20260620 19:05:02.366102 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:05:02 dist-test-slave-6wrf krb5kdc[3342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:05:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:05:02.510962 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:35543
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:35543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:02.625094 3355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:02.625437 3355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:02.625519 3355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:02.630211 3355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:02.630330 3355 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:05:02.630363 3355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:02.630410 3355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:02.630448 3355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:02.630477 3355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:02.636544 3355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:35543
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:35543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:02.637889 3355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:02.639061 3355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:02.648908 3360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:02.648998 3363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:02.649163 3361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:02.649432 3355 server_base.cc:1034] running on GCE node
I20260620 19:05:02.650487 3355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:02.652715 3355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:02.654040 3355 hybrid_clock.cc:648] HybridClock initialized: now 1781982302653985 us; error 69 us; skew 500 ppm
Jun 20 19:05:02 dist-test-slave-6wrf krb5kdc[3342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982302, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:02.657088 3355 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:05:02.658502 3355 webserver.cc:460] Webserver started at http://127.1.154.190:39085/ using document root <none> and password file <none>
I20260620 19:05:02.659237 3355 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:02.659308 3355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:02.659492 3355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:02.661758 3355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "4be88a323bb44a3d8a01d78aa908736c"
format_stamp: "Formatted at 2026-06-20 19:05:02 on dist-test-slave-6wrf"
server_key: "fef6dc3e8034c14c5813abddeb59e869"
server_key_iv: "11f73f630cdbdea5935bb0f445a77601"
server_key_version: "encryptionkey@0"
I20260620 19:05:02.662333 3355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "4be88a323bb44a3d8a01d78aa908736c"
format_stamp: "Formatted at 2026-06-20 19:05:02 on dist-test-slave-6wrf"
server_key: "fef6dc3e8034c14c5813abddeb59e869"
server_key_iv: "11f73f630cdbdea5935bb0f445a77601"
server_key_version: "encryptionkey@0"
I20260620 19:05:02.666855 3355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:05:02.669447 3370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:02.670987 3355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:05:02.671161 3355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "4be88a323bb44a3d8a01d78aa908736c"
format_stamp: "Formatted at 2026-06-20 19:05:02 on dist-test-slave-6wrf"
server_key: "fef6dc3e8034c14c5813abddeb59e869"
server_key_iv: "11f73f630cdbdea5935bb0f445a77601"
server_key_version: "encryptionkey@0"
I20260620 19:05:02.671290 3355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:02.687203 3355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:02.691862 3355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:02.692128 3355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:02.701984 3355 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:35543
I20260620 19:05:02.702023 3432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:35543 every 8 connection(s)
I20260620 19:05:02.702991 3355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:02.705807 3433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:02.709462 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 3355
I20260620 19:05:02.709581 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:02.709888 1642 external_mini_cluster.cc:1428] Setting key d4dcf614aa1eeb66723981f7c173c243
I20260620 19:05:02.711572 3433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c: Bootstrap starting.
I20260620 19:05:02.713759 3433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:02.714473 3433 log.cc:824] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:02.716250 3433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c: No bootstrap required, opened a new log
Jun 20 19:05:02 dist-test-slave-6wrf krb5kdc[3342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:02.719551 3433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be88a323bb44a3d8a01d78aa908736c" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35543 } }
I20260620 19:05:02.719790 3433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:02.719868 3433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4be88a323bb44a3d8a01d78aa908736c, State: Initialized, Role: FOLLOWER
I20260620 19:05:02.720353 3433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be88a323bb44a3d8a01d78aa908736c" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35543 } }
I20260620 19:05:02.720504 3433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:02.720618 3433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:02.720736 3433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:02.721655 3433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be88a323bb44a3d8a01d78aa908736c" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35543 } }
I20260620 19:05:02.722039 3433 leader_election.cc:302] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4be88a323bb44a3d8a01d78aa908736c; no voters:
I20260620 19:05:02.722402 3433 leader_election.cc:288] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:02.722543 3438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:02.722738 3438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [term 1 LEADER]: Becoming Leader. State: Replica: 4be88a323bb44a3d8a01d78aa908736c, State: Running, Role: LEADER
I20260620 19:05:02.723053 3438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be88a323bb44a3d8a01d78aa908736c" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35543 } }
I20260620 19:05:02.723687 3433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:02.726182 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:02.711266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.1.154.190:35543)
0620 19:05:02.711761 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:02.711777 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:02.711823 (+ 46us) server_negotiation.cc:407] Connection header received
0620 19:05:02.712484 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:02.712501 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:02.712829 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:02.713179 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:02.714030 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:02.715069 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:02.715919 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:02.716403 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:02.716570 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:02.719108 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:02.719113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:02.719126 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:02.719151 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:02.721131 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:02.721641 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:02.721648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:02.721654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:02.721797 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:02.722293 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:02.722299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:02.722301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:02.723234 (+ 933us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:02.723662 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:02.724826 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0620 19:05:02.725149 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":134,"threads_started":1}
I20260620 19:05:02.726476 3439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4be88a323bb44a3d8a01d78aa908736c. Latest consensus state: current_term: 1 leader_uuid: "4be88a323bb44a3d8a01d78aa908736c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be88a323bb44a3d8a01d78aa908736c" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35543 } } }
I20260620 19:05:02.726595 3439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:02.727385 3440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4be88a323bb44a3d8a01d78aa908736c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be88a323bb44a3d8a01d78aa908736c" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35543 } } }
I20260620 19:05:02.727492 3440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c [sys.catalog]: This master's current role is: LEADER
W20260620 19:05:02.729485 3452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:05:02.729548 3452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:05:02.729647 3454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:02.730386 3454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:02.736681 3454 catalog_manager.cc:1347] Generated new cluster ID: 49f570315e2249599342282e5a47aae8
I20260620 19:05:02.736819 3454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:02.773324 3454 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:02.774921 3454 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:02.790800 3454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c: Generated new TSK 0
I20260620 19:05:02.792205 3454 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:05:02.830622 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:35543
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:02.970355 3459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:02.970660 3459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:02.970710 3459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:02.974752 3459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:02.974822 3459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:02.974862 3459 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:05:02.974876 3459 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:05:02.974908 3459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:05:02.980607 3459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.154.190:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:02.981698 3459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:02.982684 3459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:02.994097 3465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:02.994270 3464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:02.994097 3467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:02.994982 3459 server_base.cc:1034] running on GCE node
I20260620 19:05:02.995653 3459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:02.996589 3459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:02.997962 3459 hybrid_clock.cc:648] HybridClock initialized: now 1781982302997947 us; error 39 us; skew 500 ppm
Jun 20 19:05:02 dist-test-slave-6wrf krb5kdc[3342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982302, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:03.001406 3459 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:05:03.002470 3459 webserver.cc:460] Webserver started at http://127.1.154.129:41453/ using document root <none> and password file <none>
I20260620 19:05:03.003077 3459 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:03.003176 3459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:03.003432 3459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:03.005527 3459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "7931adc7007e4c2eadc82116065512d4"
format_stamp: "Formatted at 2026-06-20 19:05:02 on dist-test-slave-6wrf"
server_key: "c6aa44e4a9d1a263cd781cbba7bf4dcf"
server_key_iv: "7e4e1b5d235c1658ff24d7f318a26d85"
server_key_version: "encryptionkey@0"
I20260620 19:05:03.006211 3459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "7931adc7007e4c2eadc82116065512d4"
format_stamp: "Formatted at 2026-06-20 19:05:02 on dist-test-slave-6wrf"
server_key: "c6aa44e4a9d1a263cd781cbba7bf4dcf"
server_key_iv: "7e4e1b5d235c1658ff24d7f318a26d85"
server_key_version: "encryptionkey@0"
I20260620 19:05:03.011896 3459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 19:05:03.014858 3474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:03.016283 3459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:03.016387 3459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "7931adc7007e4c2eadc82116065512d4"
format_stamp: "Formatted at 2026-06-20 19:05:02 on dist-test-slave-6wrf"
server_key: "c6aa44e4a9d1a263cd781cbba7bf4dcf"
server_key_iv: "7e4e1b5d235c1658ff24d7f318a26d85"
server_key_version: "encryptionkey@0"
I20260620 19:05:03.016548 3459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:03.028064 3459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:03.031713 3459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:03.031951 3459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:03.033776 3459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:03.033846 3459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:03.033913 3459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:03.033939 3459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:05:03 dist-test-slave-6wrf krb5kdc[3342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982302, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:03.045434 3459 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:39611
I20260620 19:05:03.045799 3589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:39611 every 8 connection(s)
I20260620 19:05:03.046710 3459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:05:03.050215 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.034595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:38269 (local address 127.1.154.190:35543)
0620 19:05:03.034734 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:03.034738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:03.036333 (+ 1595us) server_negotiation.cc:407] Connection header received
0620 19:05:03.037673 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:03.037677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:03.037749 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:03.037854 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:03.039088 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.039677 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.041693 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.041978 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.042114 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.045064 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:03.045069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:03.045072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:03.045101 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:03.047428 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.048091 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.048095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.048096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.048199 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.048959 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.048962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.048964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.049159 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:03.049253 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:03.049852 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 19:05:03.050021 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:05:03.050228 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 3459
I20260620 19:05:03.050333 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:05:03.050592 1642 external_mini_cluster.cc:1428] Setting key ec806ece83fb8849e75236918d9567e5
I20260620 19:05:03.051054 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.035157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35543 (local address 127.1.154.129:38269)
0620 19:05:03.035925 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:03.035999 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:03.037353 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:03.038074 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:03.038081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:03.038434 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:03.038932 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.038946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.039862 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.039869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.041537 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.041551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.042162 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.042167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.042357 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.043183 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:03.043200 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:03.044834 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:03.047635 (+ 2801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:03.047640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:03.047653 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:03.047962 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:03.048361 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:03.048364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:03.048366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:03.048803 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:03.049286 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:03.049292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:03.049535 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 19:05:03.049798 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 19:05:03.050083 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":232,"threads_started":1}
I20260620 19:05:03.052394 3590 heartbeater.cc:347] Connected to a master server at 127.1.154.190:35543
I20260620 19:05:03.052529 3590 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:03.052946 3590 heartbeater.cc:511] Master 127.1.154.190:35543 requested a full tablet report, sending...
I20260620 19:05:03.054689 3386 ts_manager.cc:194] Registered new tserver with Master: 7931adc7007e4c2eadc82116065512d4 (127.1.154.129:39611)
I20260620 19:05:03.056309 3386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:38269
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
I20260620 19:05:03.062785 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.054815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:36401 (local address 127.1.154.190:35543)
0620 19:05:03.054924 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:03.054928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:03.055232 (+ 304us) server_negotiation.cc:407] Connection header received
0620 19:05:03.055401 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:03.055403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:03.055447 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:03.055496 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:03.056085 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.056779 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.057809 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.058045 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.058158 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.059366 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:03.059370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:03.059373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:03.059395 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:03.061040 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.061500 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.061506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.061507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.061643 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.061905 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.061910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.061912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.062085 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:03.062191 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:03.062440 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 19:05:03.062573 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:05:03.062830 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.054774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35543 (local address 127.1.154.129:36401)
0620 19:05:03.055151 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:03.055167 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:03.055296 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:03.055619 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:03.055622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:03.055844 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:03.055954 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.055961 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.056943 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.056946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.057693 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.057703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.058161 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.058164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.058253 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.058847 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:03.058859 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:03.059236 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:03.061162 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:03.061167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:03.061169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:03.061399 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:03.061722 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:03.061726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:03.061728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:03.061814 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:03.062182 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:03.062185 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:03.062281 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 19:05:03.062391 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 19:05:03.062581 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:05:03.082453 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:35543
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:03.208431 3599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:03.208699 3599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:03.208748 3599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:03.212857 3599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:03.212932 3599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:03.212975 3599 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:05:03.212988 3599 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:05:03.213020 3599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:03.218286 3599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.154.190:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:03.219297 3599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:03.220316 3599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:03.230336 3605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:03.230336 3607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:03.230587 3604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:03.230791 3599 server_base.cc:1034] running on GCE node
I20260620 19:05:03.231230 3599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:03.231858 3599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:03.235162 3599 hybrid_clock.cc:648] HybridClock initialized: now 1781982303235125 us; error 82 us; skew 500 ppm
Jun 20 19:05:03 dist-test-slave-6wrf krb5kdc[3342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:03.238262 3599 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:05:03.239169 3599 webserver.cc:460] Webserver started at http://127.1.154.130:40043/ using document root <none> and password file <none>
I20260620 19:05:03.239684 3599 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:03.239740 3599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:03.239897 3599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:03.241958 3599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "1f65966760694bbaa8167ccfaa2ae318"
format_stamp: "Formatted at 2026-06-20 19:05:03 on dist-test-slave-6wrf"
server_key: "187470f66ccae33db10c49a409c1bbf1"
server_key_iv: "4f3bf239c7342f46deaf46ecc44382ba"
server_key_version: "encryptionkey@0"
I20260620 19:05:03.242466 3599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "1f65966760694bbaa8167ccfaa2ae318"
format_stamp: "Formatted at 2026-06-20 19:05:03 on dist-test-slave-6wrf"
server_key: "187470f66ccae33db10c49a409c1bbf1"
server_key_iv: "4f3bf239c7342f46deaf46ecc44382ba"
server_key_version: "encryptionkey@0"
I20260620 19:05:03.246196 3599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:05:03.248554 3614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:03.250443 3599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:03.250645 3599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "1f65966760694bbaa8167ccfaa2ae318"
format_stamp: "Formatted at 2026-06-20 19:05:03 on dist-test-slave-6wrf"
server_key: "187470f66ccae33db10c49a409c1bbf1"
server_key_iv: "4f3bf239c7342f46deaf46ecc44382ba"
server_key_version: "encryptionkey@0"
I20260620 19:05:03.250988 3599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:03.278914 3599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:03.282931 3599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:03.283177 3599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:03.284665 3599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:03.284714 3599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:03.284756 3599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:03.284771 3599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:03.296429 3599 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:46493
I20260620 19:05:03.296725 3729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:46493 every 8 connection(s)
Jun 20 19:05:03 dist-test-slave-6wrf krb5kdc[3342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:03.297510 3599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:03.301829 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 3599
I20260620 19:05:03.301942 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:05:03.301999 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.285819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:59921 (local address 127.1.154.190:35543)
0620 19:05:03.286004 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:03.286008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:03.286943 (+ 935us) server_negotiation.cc:407] Connection header received
0620 19:05:03.288523 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:03.288527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:03.288582 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:03.288673 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:03.290401 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.291272 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.293261 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.293669 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.293875 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.298041 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:03.298045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:03.298048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:03.298079 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:03.299578 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.300119 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.300122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.300123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.300202 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.300596 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.300598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.300600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.300964 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:03.301137 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:03.301714 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 19:05:03.301826 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 19:05:03.302244 1642 external_mini_cluster.cc:1428] Setting key 325e5adc46e0c9179b26638e23eb91db
I20260620 19:05:03.302876 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.286297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35543 (local address 127.1.154.130:59921)
0620 19:05:03.286783 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:03.286842 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:03.288290 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:03.288885 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:03.288896 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:03.289448 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:03.290228 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.290251 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.291395 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.291400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.293024 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.293037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.294028 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.294035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.294600 (+ 565us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.295717 (+ 1117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:03.295733 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:03.297802 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:03.299713 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:03.299718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:03.299728 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:03.299989 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:03.300307 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:03.300309 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:03.300311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:03.300467 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:03.301136 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:03.301141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:03.301365 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 19:05:03.301574 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 19:05:03.301943 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":102,"threads_started":1}
I20260620 19:05:03.304159 3730 heartbeater.cc:347] Connected to a master server at 127.1.154.190:35543
I20260620 19:05:03.304288 3730 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:03.304803 3730 heartbeater.cc:511] Master 127.1.154.190:35543 requested a full tablet report, sending...
I20260620 19:05:03.305867 3386 ts_manager.cc:194] Registered new tserver with Master: 1f65966760694bbaa8167ccfaa2ae318 (127.1.154.130:46493)
I20260620 19:05:03.306584 3386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:59921
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
I20260620 19:05:03.320369 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.306316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35543 (local address 127.1.154.130:34737)
0620 19:05:03.310092 (+ 3776us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:03.310108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:03.310202 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:03.311019 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:03.311022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:03.311201 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:03.311296 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.311302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.312555 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.312565 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.313366 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.313379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.314096 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.314100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.314510 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.315393 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:03.315409 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:03.315865 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:03.318511 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:03.318515 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:03.318518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:03.318882 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:03.319234 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:03.319238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:03.319240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:03.319328 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:03.319782 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:03.319786 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:03.319888 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 19:05:03.320002 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 19:05:03.320162 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3341,"thread_start_us":88,"threads_started":1}
I20260620 19:05:03.320403 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.306378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:34737 (local address 127.1.154.190:35543)
0620 19:05:03.306505 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:03.306510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:03.310337 (+ 3827us) server_negotiation.cc:407] Connection header received
0620 19:05:03.310595 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:03.310601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:03.310835 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:03.310960 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:03.311419 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.312289 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.313510 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.313893 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.314003 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.316094 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:03.316098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:03.316100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:03.316128 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:03.318377 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.319040 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.319043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.319045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.319134 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.319445 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.319449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.319451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.319679 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:03.319798 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:03.320062 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 19:05:03.320238 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:05:03.338676 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:35543
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:03.469529 3739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:03.469851 3739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:03.469899 3739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:03.473946 3739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:03.474009 3739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:03.474049 3739 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:05:03.474063 3739 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:05:03.474097 3739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:03.479468 3739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.154.190:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:03.480511 3739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:03.481513 3739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:03.492039 3745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:03.492058 3744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:03.492280 3747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:03.492525 3739 server_base.cc:1034] running on GCE node
I20260620 19:05:03.492929 3739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:03.493631 3739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:03.494834 3739 hybrid_clock.cc:648] HybridClock initialized: now 1781982303494812 us; error 60 us; skew 500 ppm
Jun 20 19:05:03 dist-test-slave-6wrf krb5kdc[3342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:03.497956 3739 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:05:03.498858 3739 webserver.cc:460] Webserver started at http://127.1.154.131:39019/ using document root <none> and password file <none>
I20260620 19:05:03.499428 3739 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:03.499490 3739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:03.499691 3739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:03.501854 3739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "b83ba29e68cf43759bfb8117d6be5b8e"
format_stamp: "Formatted at 2026-06-20 19:05:03 on dist-test-slave-6wrf"
server_key: "e9c8553dad9aab260f5b74f81dd6f672"
server_key_iv: "2cf6267c6a41bf63d247b0582d338824"
server_key_version: "encryptionkey@0"
I20260620 19:05:03.502415 3739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "b83ba29e68cf43759bfb8117d6be5b8e"
format_stamp: "Formatted at 2026-06-20 19:05:03 on dist-test-slave-6wrf"
server_key: "e9c8553dad9aab260f5b74f81dd6f672"
server_key_iv: "2cf6267c6a41bf63d247b0582d338824"
server_key_version: "encryptionkey@0"
I20260620 19:05:03.507465 3739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 19:05:03.510497 3754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:03.512576 3739 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:05:03.512677 3739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "b83ba29e68cf43759bfb8117d6be5b8e"
format_stamp: "Formatted at 2026-06-20 19:05:03 on dist-test-slave-6wrf"
server_key: "e9c8553dad9aab260f5b74f81dd6f672"
server_key_iv: "2cf6267c6a41bf63d247b0582d338824"
server_key_version: "encryptionkey@0"
I20260620 19:05:03.512835 3739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:03.530663 3739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:03.536481 3739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:03.536785 3739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:03.538928 3739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:03.539019 3739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:03.539091 3739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:03.539143 3739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:05:03 dist-test-slave-6wrf krb5kdc[3342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:03.554284 3739 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:44421
I20260620 19:05:03.554340 3869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:44421 every 8 connection(s)
I20260620 19:05:03.555486 3739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:03.557037 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.540032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:38815 (local address 127.1.154.190:35543)
0620 19:05:03.540220 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:03.540226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:03.541172 (+ 946us) server_negotiation.cc:407] Connection header received
0620 19:05:03.542200 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:03.542204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:03.542262 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:03.542367 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:03.544004 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.544726 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.546210 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.546536 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.546683 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.550782 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:03.550786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:03.550789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:03.550823 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:03.553695 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.554393 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.554397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.554399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.554492 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.555044 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.555052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.555055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.555469 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:03.555632 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:03.556732 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0620 19:05:03.556859 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 19:05:03.558001 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 3739
I20260620 19:05:03.558174 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:05:03.558643 1642 external_mini_cluster.cc:1428] Setting key c3e27f1787b0810c25715ed237fcdc58
I20260620 19:05:03.558640 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.540453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35543 (local address 127.1.154.131:38815)
0620 19:05:03.540965 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:03.541014 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:03.541981 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:03.542545 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:03.542553 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:03.543086 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:03.543845 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.543866 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.544867 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.544873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.546082 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.546091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.546718 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.546722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.546918 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.547654 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:03.547678 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:03.550354 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:03.553840 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:03.553851 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:03.553870 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:03.554263 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:03.554596 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:03.554599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:03.554601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:03.554782 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:03.555647 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:03.555659 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:03.556275 (+ 616us) client_negotiation.cc:769] Sending connection context
0620 19:05:03.556652 (+ 377us) client_negotiation.cc:240] Negotiation successful
0620 19:05:03.557147 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":147,"threads_started":1}
I20260620 19:05:03.560406 3870 heartbeater.cc:347] Connected to a master server at 127.1.154.190:35543
I20260620 19:05:03.560781 3870 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:03.561717 3870 heartbeater.cc:511] Master 127.1.154.190:35543 requested a full tablet report, sending...
I20260620 19:05:03.562901 3387 ts_manager.cc:194] Registered new tserver with Master: b83ba29e68cf43759bfb8117d6be5b8e (127.1.154.131:44421)
I20260620 19:05:03.563759 3387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:38815
I20260620 19:05:03.571707 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.563359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35543 (local address 127.1.154.131:52695)
0620 19:05:03.563644 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:03.563656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:03.563741 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:03.564081 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:03.564085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:03.564327 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:03.564452 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.564458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.565185 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.565189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.565874 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.565882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.566410 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.566413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.566524 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.567169 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:03.567181 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:03.567545 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:03.570222 (+ 2677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:03.570225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:03.570227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:03.570446 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:03.570738 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:03.570740 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:03.570742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:03.570824 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:03.571243 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:03.571245 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:03.571328 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 19:05:03.571427 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 19:05:03.571565 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
I20260620 19:05:03.572244 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.563405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:52695 (local address 127.1.154.190:35543)
0620 19:05:03.563530 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:03.563535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:03.563725 (+ 190us) server_negotiation.cc:407] Connection header received
0620 19:05:03.563815 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:03.563819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:03.563861 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:03.563941 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:03.564564 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.565066 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.565990 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.566220 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.566355 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.567880 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:03.567884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:03.567887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:03.567928 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:03.570108 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.570549 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.570552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.570553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.570631 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.570960 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.570963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.570964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.571141 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:03.571224 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:03.571785 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 19:05:03.572054 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:05:03.574573 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:05:03 dist-test-slave-6wrf krb5kdc[3342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:05:03 dist-test-slave-6wrf krb5kdc[3342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:03.607070 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.597146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.1.154.190:35543)
0620 19:05:03.597370 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:03.597374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:03.597396 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:05:03.597452 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:03.597455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:03.597503 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:03.597688 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:03.598511 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.599357 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.600407 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.600727 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.600864 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.603619 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:03.603623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:03.603625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:03.603655 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:03.605196 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.605768 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.605773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.605775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.605856 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.606166 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.606169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.606171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.606365 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:03.606521 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:03.606767 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 19:05:03.606890 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 19:05:03.611219 3387 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:05:03.614327 3387 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:03.630700 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.624134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:46493 (local address 127.0.0.1:33328)
0620 19:05:03.624355 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:03.624369 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:03.624460 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:03.625494 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:03.625501 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:03.625526 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:03.625770 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.625778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.627509 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.627516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.628940 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.628949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.630145 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.630148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.630256 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.630299 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 19:05:03.630405 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 19:05:03.630539 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":65,"thread_start_us":87,"threads_started":1}
I20260620 19:05:03.630748 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.624691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.1.154.130:46493)
0620 19:05:03.624992 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:03.625004 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:03.625028 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:05:03.625106 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:03.625112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:03.625310 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:03.626120 (+ 810us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:03.626242 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.627388 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.629096 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.629975 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.630145 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.630442 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:03.630512 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 19:05:03.630625 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":126,"threads_started":1}
I20260620 19:05:03.631803 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.625314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.1.154.129:39611)
0620 19:05:03.625739 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:03.625747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:03.625812 (+ 65us) server_negotiation.cc:407] Connection header received
0620 19:05:03.625923 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:03.625930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:03.626191 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:03.626761 (+ 570us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:03.626930 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.628068 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.630077 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.630991 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.631111 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.631249 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:03.631547 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 19:05:03.631662 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":145,"threads_started":1}
I20260620 19:05:03.631870 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.624562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:39611 (local address 127.0.0.1:45140)
0620 19:05:03.625032 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:03.625129 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:03.625379 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:03.626373 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:03.626379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:03.626399 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:03.626571 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.626581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.628221 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.628225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.629930 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.629938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.631202 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.631205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.631309 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.631335 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 19:05:03.631490 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 19:05:03.631620 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":120,"threads_started":1}
I20260620 19:05:03.631920 3888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.623917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44754 (local address 127.1.154.131:44421)
0620 19:05:03.625168 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:03.625184 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:03.625202 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:05:03.625262 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:03.625266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:03.625488 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:03.625779 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:03.626076 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.627244 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.628940 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.629975 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.630145 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.630441 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:03.631470 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0620 19:05:03.631613 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1167,"thread_start_us":154,"threads_started":1}
I20260620 19:05:03.631603 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.623786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:44421 (local address 127.0.0.1:44754)
0620 19:05:03.624211 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:03.624254 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:03.624385 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:03.625646 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:03.625649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:03.625662 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:03.625950 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.625965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.627383 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.627386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.628781 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.628791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.631163 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.631167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.631262 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.631290 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 19:05:03.631376 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:05:03.631439 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":218,"thread_start_us":116,"threads_started":1}
I20260620 19:05:03.636085 3524 tablet_service.cc:1467] Processing CreateTablet for tablet 157b97251a84476cafa09501a91bea65 (DEFAULT_TABLE table=test-table [id=7e3939f7749e42cca7c3b10d84631ec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:03.635977 3664 tablet_service.cc:1467] Processing CreateTablet for tablet 157b97251a84476cafa09501a91bea65 (DEFAULT_TABLE table=test-table [id=7e3939f7749e42cca7c3b10d84631ec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:03.637252 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 157b97251a84476cafa09501a91bea65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:03.637410 3664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 157b97251a84476cafa09501a91bea65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:03.637379 3804 tablet_service.cc:1467] Processing CreateTablet for tablet 157b97251a84476cafa09501a91bea65 (DEFAULT_TABLE table=test-table [id=7e3939f7749e42cca7c3b10d84631ec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:03.638710 3804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 157b97251a84476cafa09501a91bea65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:03.643610 3893 tablet_bootstrap.cc:492] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4: Bootstrap starting.
I20260620 19:05:03.646337 3893 tablet_bootstrap.cc:654] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:03.646605 3894 tablet_bootstrap.cc:492] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318: Bootstrap starting.
I20260620 19:05:03.647780 3895 tablet_bootstrap.cc:492] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e: Bootstrap starting.
I20260620 19:05:03.648123 3893 log.cc:824] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:03.648816 3894 tablet_bootstrap.cc:654] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:03.649555 3894 log.cc:824] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:03.650784 3893 tablet_bootstrap.cc:492] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4: No bootstrap required, opened a new log
I20260620 19:05:03.650990 3893 ts_tablet_manager.cc:1397] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 19:05:03.651201 3895 tablet_bootstrap.cc:654] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:03.651669 3894 tablet_bootstrap.cc:492] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318: No bootstrap required, opened a new log
I20260620 19:05:03.651854 3894 ts_tablet_manager.cc:1397] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:05:03.652323 3895 log.cc:824] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:03.654511 3895 tablet_bootstrap.cc:492] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e: No bootstrap required, opened a new log
I20260620 19:05:03.654752 3895 ts_tablet_manager.cc:1397] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260620 19:05:03.655784 3893 raft_consensus.cc:348] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } }
I20260620 19:05:03.655982 3894 raft_consensus.cc:348] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } }
I20260620 19:05:03.656101 3893 raft_consensus.cc:374] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:03.656203 3893 raft_consensus.cc:729] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7931adc7007e4c2eadc82116065512d4, State: Initialized, Role: FOLLOWER
I20260620 19:05:03.656229 3894 raft_consensus.cc:374] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:03.656284 3894 raft_consensus.cc:729] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f65966760694bbaa8167ccfaa2ae318, State: Initialized, Role: FOLLOWER
I20260620 19:05:03.656701 3894 consensus_queue.cc:260] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } }
I20260620 19:05:03.656701 3893 consensus_queue.cc:260] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } }
I20260620 19:05:03.657589 3590 heartbeater.cc:503] Master 127.1.154.190:35543 was elected leader, sending a full tablet report...
I20260620 19:05:03.657625 3730 heartbeater.cc:503] Master 127.1.154.190:35543 was elected leader, sending a full tablet report...
I20260620 19:05:03.657727 3893 ts_tablet_manager.cc:1428] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 19:05:03.657724 3894 ts_tablet_manager.cc:1428] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:05:03.660310 3895 raft_consensus.cc:348] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } }
I20260620 19:05:03.660657 3895 raft_consensus.cc:374] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:03.660717 3895 raft_consensus.cc:729] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83ba29e68cf43759bfb8117d6be5b8e, State: Initialized, Role: FOLLOWER
I20260620 19:05:03.661172 3895 consensus_queue.cc:260] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } }
I20260620 19:05:03.661859 3870 heartbeater.cc:503] Master 127.1.154.190:35543 was elected leader, sending a full tablet report...
I20260620 19:05:03.663268 3895 ts_tablet_manager.cc:1428] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260620 19:05:03.751127 3900 raft_consensus.cc:482] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:03.751394 3900 raft_consensus.cc:504] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } }
I20260620 19:05:03.752672 3900 leader_election.cc:288] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b83ba29e68cf43759bfb8117d6be5b8e (127.1.154.131:44421), 7931adc7007e4c2eadc82116065512d4 (127.1.154.129:39611)
I20260620 19:05:03.758873 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.753087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:44421 (local address 127.1.154.130:59089)
0620 19:05:03.753209 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:03.753229 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:03.753333 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:03.753862 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:03.753867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:03.753885 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:03.754019 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.754024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.755135 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.755140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.756662 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.756667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.757947 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.757954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.758214 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.758261 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 19:05:03.758463 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 19:05:03.758615 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":156}
I20260620 19:05:03.759104 3888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.753388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:59089 (local address 127.1.154.131:44421)
0620 19:05:03.753549 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:03.753555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:03.753571 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:05:03.753657 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:03.753663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:03.753717 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:03.753827 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:03.754156 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.754971 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.757001 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.757710 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.757841 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.758011 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:03.758651 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 19:05:03.758773 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 19:05:03.759647 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.753499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:35921 (local address 127.1.154.129:39611)
0620 19:05:03.753616 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:03.753622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:03.753786 (+ 164us) server_negotiation.cc:407] Connection header received
0620 19:05:03.753923 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:03.753961 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:03.754008 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:03.754069 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:03.754340 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.755050 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.756972 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.757695 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.757841 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.758011 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:03.759264 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0620 19:05:03.759416 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:05:03.760053 3824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157b97251a84476cafa09501a91bea65" candidate_uuid: "1f65966760694bbaa8167ccfaa2ae318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" is_pre_election: true
I20260620 19:05:03.760162 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.753372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:39611 (local address 127.1.154.130:35921)
0620 19:05:03.753690 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:03.753702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:03.753788 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:03.754113 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:03.754119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:03.754140 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:03.754244 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.754249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.755154 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.755157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.756655 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.756666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.757947 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.757954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.759035 (+ 1081us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.759065 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 19:05:03.759897 (+ 832us) client_negotiation.cc:240] Negotiation successful
0620 19:05:03.760014 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":212,"thread_start_us":103,"threads_started":1}
I20260620 19:05:03.761088 3824 raft_consensus.cc:2454] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f65966760694bbaa8167ccfaa2ae318 in term 0.
I20260620 19:05:03.761758 3617 leader_election.cc:302] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f65966760694bbaa8167ccfaa2ae318, b83ba29e68cf43759bfb8117d6be5b8e; no voters:
I20260620 19:05:03.762084 3900 raft_consensus.cc:2781] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:03.762193 3900 raft_consensus.cc:482] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:03.762102 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157b97251a84476cafa09501a91bea65" candidate_uuid: "1f65966760694bbaa8167ccfaa2ae318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7931adc7007e4c2eadc82116065512d4" is_pre_election: true
I20260620 19:05:03.762509 3900 raft_consensus.cc:3037] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:03.762681 3544 raft_consensus.cc:2454] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f65966760694bbaa8167ccfaa2ae318 in term 0.
I20260620 19:05:03.763760 3900 raft_consensus.cc:504] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } }
I20260620 19:05:03.764134 3900 leader_election.cc:288] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [CANDIDATE]: Term 1 election: Requested vote from peers b83ba29e68cf43759bfb8117d6be5b8e (127.1.154.131:44421), 7931adc7007e4c2eadc82116065512d4 (127.1.154.129:39611)
I20260620 19:05:03.765100 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157b97251a84476cafa09501a91bea65" candidate_uuid: "1f65966760694bbaa8167ccfaa2ae318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7931adc7007e4c2eadc82116065512d4"
I20260620 19:05:03.765251 3544 raft_consensus.cc:3037] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:03.765295 3824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157b97251a84476cafa09501a91bea65" candidate_uuid: "1f65966760694bbaa8167ccfaa2ae318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83ba29e68cf43759bfb8117d6be5b8e"
I20260620 19:05:03.765414 3824 raft_consensus.cc:3037] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:03.766472 3544 raft_consensus.cc:2454] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f65966760694bbaa8167ccfaa2ae318 in term 1.
I20260620 19:05:03.766489 3824 raft_consensus.cc:2454] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f65966760694bbaa8167ccfaa2ae318 in term 1.
I20260620 19:05:03.766808 3617 leader_election.cc:302] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f65966760694bbaa8167ccfaa2ae318, 7931adc7007e4c2eadc82116065512d4; no voters:
I20260620 19:05:03.767017 3900 raft_consensus.cc:2781] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:03.767508 3900 raft_consensus.cc:686] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [term 1 LEADER]: Becoming Leader. State: Replica: 1f65966760694bbaa8167ccfaa2ae318, State: Running, Role: LEADER
I20260620 19:05:03.767979 3900 consensus_queue.cc:237] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } }
I20260620 19:05:03.772889 3386 catalog_manager.cc:5495] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f65966760694bbaa8167ccfaa2ae318 (127.1.154.130). New cstate: current_term: 1 leader_uuid: "1f65966760694bbaa8167ccfaa2ae318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } health_report { overall_health: HEALTHY } } }
W20260620 19:05:03.798483 3591 tablet.cc:2369] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:05:03.799257 3731 tablet.cc:2369] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:05:03.807111 3871 tablet.cc:2369] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 19:05:03 dist-test-slave-6wrf krb5kdc[3342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982302, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:03.821362 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.807863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58062 (local address 127.1.154.190:35543)
0620 19:05:03.808130 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:03.808134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:03.808152 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:05:03.808208 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:03.808211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:03.808262 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:03.808393 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:03.809907 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.810442 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.811744 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.811981 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.812459 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.816318 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:03.816323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:03.816325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:03.816363 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:03.818380 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.819439 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.819443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.819445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.819569 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:03.820076 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:03.820079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:03.820080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:03.820375 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:03.820487 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:03.821009 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 19:05:03.821189 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 19:05:03.821955 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:03.807633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35543 (local address 127.0.0.1:58062)
0620 19:05:03.807973 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:03.808001 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:03.808094 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:03.808496 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:03.808499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:03.809479 (+ 980us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:03.809664 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.809673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.810575 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.810578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.811211 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:03.811220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:03.812516 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:03.812521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:03.812651 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:03.813776 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:03.813793 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:03.816041 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:03.818595 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:03.818617 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:03.818629 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:03.819193 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:03.819777 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:03.819780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:03.819782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:03.819904 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:03.820572 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:03.820580 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:03.820726 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 19:05:03.820862 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 19:05:03.821341 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":143,"threads_started":1}
I20260620 19:05:03.828619 3386 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:58062:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:05:03.829460 3386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:03.836505 3524 tablet_service.cc:1467] Processing CreateTablet for tablet 8ceb0f2f301c4867a0816121cae9aedb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bde0b7c296bf4a42a5c2015909077210]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:05:03.836489 3664 tablet_service.cc:1467] Processing CreateTablet for tablet 8ceb0f2f301c4867a0816121cae9aedb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bde0b7c296bf4a42a5c2015909077210]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:05:03.837338 3664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ceb0f2f301c4867a0816121cae9aedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:03.836962 3804 tablet_service.cc:1467] Processing CreateTablet for tablet 8ceb0f2f301c4867a0816121cae9aedb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bde0b7c296bf4a42a5c2015909077210]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:05:03.837385 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ceb0f2f301c4867a0816121cae9aedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:03.837647 3804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ceb0f2f301c4867a0816121cae9aedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:03.840035 3895 tablet_bootstrap.cc:492] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e: Bootstrap starting.
I20260620 19:05:03.840369 3893 tablet_bootstrap.cc:492] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4: Bootstrap starting.
I20260620 19:05:03.841120 3895 tablet_bootstrap.cc:654] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:03.841555 3893 tablet_bootstrap.cc:654] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:03.841822 3894 tablet_bootstrap.cc:492] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318: Bootstrap starting.
I20260620 19:05:03.842484 3895 tablet_bootstrap.cc:492] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e: No bootstrap required, opened a new log
I20260620 19:05:03.842555 3895 ts_tablet_manager.cc:1397] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 19:05:03.842768 3894 tablet_bootstrap.cc:654] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:03.842823 3893 tablet_bootstrap.cc:492] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4: No bootstrap required, opened a new log
I20260620 19:05:03.842900 3893 ts_tablet_manager.cc:1397] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:03.843062 3895 raft_consensus.cc:348] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:03.843181 3895 raft_consensus.cc:374] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:03.843214 3895 raft_consensus.cc:729] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83ba29e68cf43759bfb8117d6be5b8e, State: Initialized, Role: FOLLOWER
I20260620 19:05:03.843237 3893 raft_consensus.cc:348] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:03.843356 3893 raft_consensus.cc:374] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:03.843353 3895 consensus_queue.cc:260] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:03.843400 3893 raft_consensus.cc:729] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7931adc7007e4c2eadc82116065512d4, State: Initialized, Role: FOLLOWER
I20260620 19:05:03.843601 3893 consensus_queue.cc:260] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:03.843595 3901 tablet_replica.cc:439] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } }
I20260620 19:05:03.843745 3901 tablet_replica.cc:442] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:03.843866 3899 tablet_replica.cc:439] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } }
I20260620 19:05:03.844003 3899 tablet_replica.cc:442] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:03.844074 3894 tablet_bootstrap.cc:492] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318: No bootstrap required, opened a new log
I20260620 19:05:03.844156 3894 ts_tablet_manager.cc:1397] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:05:03.844548 3895 ts_tablet_manager.cc:1428] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:05:03.844628 3894 raft_consensus.cc:348] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:03.844750 3894 raft_consensus.cc:374] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:03.844784 3894 raft_consensus.cc:729] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f65966760694bbaa8167ccfaa2ae318, State: Initialized, Role: FOLLOWER
I20260620 19:05:03.844902 3894 consensus_queue.cc:260] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:03.845180 3894 ts_tablet_manager.cc:1428] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:05:03.845178 3893 ts_tablet_manager.cc:1428] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:05:03.845386 3900 tablet_replica.cc:439] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } }
I20260620 19:05:03.845503 3900 tablet_replica.cc:442] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:04.047201 3900 raft_consensus.cc:482] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:04.047410 3900 raft_consensus.cc:504] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:04.048113 3900 leader_election.cc:288] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7931adc7007e4c2eadc82116065512d4 (127.1.154.129:39611), b83ba29e68cf43759bfb8117d6be5b8e (127.1.154.131:44421)
I20260620 19:05:04.048693 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ceb0f2f301c4867a0816121cae9aedb" candidate_uuid: "1f65966760694bbaa8167ccfaa2ae318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7931adc7007e4c2eadc82116065512d4" is_pre_election: true
I20260620 19:05:04.048903 3544 raft_consensus.cc:2454] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f65966760694bbaa8167ccfaa2ae318 in term 0.
I20260620 19:05:04.048892 3824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ceb0f2f301c4867a0816121cae9aedb" candidate_uuid: "1f65966760694bbaa8167ccfaa2ae318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" is_pre_election: true
I20260620 19:05:04.049122 3824 raft_consensus.cc:2454] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f65966760694bbaa8167ccfaa2ae318 in term 0.
I20260620 19:05:04.049512 3617 leader_election.cc:302] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f65966760694bbaa8167ccfaa2ae318, b83ba29e68cf43759bfb8117d6be5b8e; no voters:
I20260620 19:05:04.049731 3900 raft_consensus.cc:2781] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:04.049800 3900 raft_consensus.cc:482] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:04.049844 3900 raft_consensus.cc:3037] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:04.051092 3900 raft_consensus.cc:504] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:04.051692 3900 leader_election.cc:288] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [CANDIDATE]: Term 1 election: Requested vote from peers 7931adc7007e4c2eadc82116065512d4 (127.1.154.129:39611), b83ba29e68cf43759bfb8117d6be5b8e (127.1.154.131:44421)
I20260620 19:05:04.052090 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ceb0f2f301c4867a0816121cae9aedb" candidate_uuid: "1f65966760694bbaa8167ccfaa2ae318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7931adc7007e4c2eadc82116065512d4"
I20260620 19:05:04.052191 3824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ceb0f2f301c4867a0816121cae9aedb" candidate_uuid: "1f65966760694bbaa8167ccfaa2ae318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83ba29e68cf43759bfb8117d6be5b8e"
I20260620 19:05:04.052249 3544 raft_consensus.cc:3037] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:04.052335 3824 raft_consensus.cc:3037] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:04.053766 3544 raft_consensus.cc:2454] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f65966760694bbaa8167ccfaa2ae318 in term 1.
I20260620 19:05:04.053766 3824 raft_consensus.cc:2454] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f65966760694bbaa8167ccfaa2ae318 in term 1.
I20260620 19:05:04.054227 3617 leader_election.cc:302] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f65966760694bbaa8167ccfaa2ae318, b83ba29e68cf43759bfb8117d6be5b8e; no voters:
I20260620 19:05:04.054487 3900 raft_consensus.cc:2781] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:04.054610 3900 raft_consensus.cc:686] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [term 1 LEADER]: Becoming Leader. State: Replica: 1f65966760694bbaa8167ccfaa2ae318, State: Running, Role: LEADER
I20260620 19:05:04.054771 3900 consensus_queue.cc:237] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:04.055876 3903 tablet_replica.cc:439] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318: TxnStatusTablet state changed. Reason: New leader 1f65966760694bbaa8167ccfaa2ae318. Latest consensus state: current_term: 1 leader_uuid: "1f65966760694bbaa8167ccfaa2ae318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } }
I20260620 19:05:04.056110 3903 tablet_replica.cc:442] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:05:04.057072 3915 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:05:04.057344 3915 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:05:04.057929 3385 catalog_manager.cc:5495] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f65966760694bbaa8167ccfaa2ae318 (127.1.154.130). New cstate: current_term: 1 leader_uuid: "1f65966760694bbaa8167ccfaa2ae318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } health_report { overall_health: UNKNOWN } } }
I20260620 19:05:04.100225 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:04.094818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45046 (local address 127.1.154.130:46493)
0620 19:05:04.095002 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:04.095008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:04.095032 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:05:04.095245 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:04.095249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:04.095335 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:04.095469 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:04.095749 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.096619 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.097927 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.098326 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.098455 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:04.098778 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:04.099584 (+ 806us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:04.099851 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:04.099961 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 19:05:04.100050 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 19:05:04.100224 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:04.094589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:46493 (local address 127.0.0.1:45046)
0620 19:05:04.094942 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:04.094958 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:04.095062 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:04.095529 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:04.095532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:04.095539 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:05:04.095639 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:04.095644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.096794 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.096798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:04.097778 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:04.097790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.098479 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.098482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:04.098558 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:04.098607 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:05:04.099820 (+ 1213us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:05:04.099852 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 19:05:04.099945 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 19:05:04.100072 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260620 19:05:04.106109 3544 raft_consensus.cc:1261] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Refusing update from remote peer 1f65966760694bbaa8167ccfaa2ae318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:04.106530 3824 raft_consensus.cc:1261] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Refusing update from remote peer 1f65966760694bbaa8167ccfaa2ae318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:04.106940 3900 consensus_queue.cc:1035] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:04.107308 3903 consensus_queue.cc:1035] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:04.111855 3901 tablet_replica.cc:439] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e: TxnStatusTablet state changed. Reason: New leader 1f65966760694bbaa8167ccfaa2ae318. Latest consensus state: current_term: 1 leader_uuid: "1f65966760694bbaa8167ccfaa2ae318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } }
I20260620 19:05:04.112015 3901 tablet_replica.cc:442] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:04.113541 3899 tablet_replica.cc:439] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4: TxnStatusTablet state changed. Reason: New leader 1f65966760694bbaa8167ccfaa2ae318. Latest consensus state: current_term: 1 leader_uuid: "1f65966760694bbaa8167ccfaa2ae318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } }
I20260620 19:05:04.113745 3899 tablet_replica.cc:442] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:04.117230 3903 tablet_replica.cc:439] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f65966760694bbaa8167ccfaa2ae318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } }
I20260620 19:05:04.117367 3903 tablet_replica.cc:442] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:05:04.118013 3903 tablet_replica.cc:439] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f65966760694bbaa8167ccfaa2ae318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } }
I20260620 19:05:04.118119 3903 tablet_replica.cc:442] T 8ceb0f2f301c4867a0816121cae9aedb P 1f65966760694bbaa8167ccfaa2ae318: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:05:04.118937 3899 tablet_replica.cc:439] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f65966760694bbaa8167ccfaa2ae318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } }
I20260620 19:05:04.119076 3899 tablet_replica.cc:442] T 8ceb0f2f301c4867a0816121cae9aedb P 7931adc7007e4c2eadc82116065512d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:04.120566 3901 tablet_replica.cc:439] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f65966760694bbaa8167ccfaa2ae318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } } }
I20260620 19:05:04.120690 3901 tablet_replica.cc:442] T 8ceb0f2f301c4867a0816121cae9aedb P b83ba29e68cf43759bfb8117d6be5b8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:04.122545 3919 mvcc.cc:205] Tried to move back new op lower bound from 7298999517611126784 to 7298999517410222080. Current Snapshot: MvccSnapshot[applied={T|T < 7298999517611126784}]
I20260620 19:05:04.130003 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:04.125531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.1.154.130:46493)
0620 19:05:04.125706 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:04.125711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:04.125725 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:05:04.125768 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:04.125771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:04.125839 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:04.125926 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:04.126521 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.127250 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.128585 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.128975 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.129121 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:04.129444 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:04.129534 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:04.129578 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:04.129778 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 19:05:04.129856 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:05:04.154253 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:04.149200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:46493 (local address 127.1.154.130:50699)
0620 19:05:04.149907 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:04.149922 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:04.150091 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:04.150334 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:04.150336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:04.150356 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:05:04.150490 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:04.150501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.151745 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.151748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:04.152598 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:04.152607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.153279 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.153282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:04.153361 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:04.153413 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:05:04.153885 (+ 472us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:05:04.153891 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:05:04.153965 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:05:04.154032 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":3,"thread_start_us":143,"threads_started":1}
I20260620 19:05:04.154296 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:04.149331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:50699 (local address 127.1.154.130:46493)
0620 19:05:04.149493 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:04.149499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:04.150025 (+ 526us) server_negotiation.cc:407] Connection header received
0620 19:05:04.150195 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:04.150197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:04.150244 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:04.150327 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:04.150688 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.151533 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.152727 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.153104 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.153265 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:04.153673 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:04.153787 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:04.153874 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:04.153998 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 19:05:04.154070 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":52}
I20260620 19:05:04.164327 3544 raft_consensus.cc:1261] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Refusing update from remote peer 1f65966760694bbaa8167ccfaa2ae318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:04.164361 3824 raft_consensus.cc:1261] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Refusing update from remote peer 1f65966760694bbaa8167ccfaa2ae318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:04.164860 3931 consensus_queue.cc:1035] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:04.165269 3900 consensus_queue.cc:1035] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:04.169154 3918 mvcc.cc:205] Tried to move back new op lower bound from 7298999517851594752 to 7298999516238753792. Current Snapshot: MvccSnapshot[applied={T|T < 7298999517851594752}]
I20260620 19:05:04.173132 3921 mvcc.cc:205] Tried to move back new op lower bound from 7298999517851594752 to 7298999516238753792. Current Snapshot: MvccSnapshot[applied={T|T < 7298999517851594752}]
I20260620 19:05:04.216028 3385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:04.216243 3385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:04.219892 3385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c: Sending DeleteTablet for 3 replicas of tablet 157b97251a84476cafa09501a91bea65
I20260620 19:05:04.220830 3804 tablet_service.cc:1514] Processing DeleteTablet for tablet 157b97251a84476cafa09501a91bea65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:04 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:44754
I20260620 19:05:04.220808 3524 tablet_service.cc:1514] Processing DeleteTablet for tablet 157b97251a84476cafa09501a91bea65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:04 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:45140
I20260620 19:05:04.220808 3663 tablet_service.cc:1514] Processing DeleteTablet for tablet 157b97251a84476cafa09501a91bea65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:04 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:33328
I20260620 19:05:04.221426 3936 tablet_replica.cc:330] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e: stopping tablet replica
I20260620 19:05:04.221477 3937 tablet_replica.cc:330] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4: stopping tablet replica
I20260620 19:05:04.221951 3937 raft_consensus.cc:2229] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:04.222016 3936 raft_consensus.cc:2229] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:04.222258 3937 raft_consensus.cc:2258] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:04.222483 3386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:05:04.222779 3938 tablet_replica.cc:330] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318: stopping tablet replica
W20260620 19:05:04.222945 3386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:04.222498 3936 raft_consensus.cc:2258] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:04.223119 3938 raft_consensus.cc:2229] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:05:04.223575 3938 raft_consensus.cc:2258] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:04.225209 3936 ts_tablet_manager.cc:1905] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:04.225306 3937 ts_tablet_manager.cc:1905] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:04.228852 3804 tablet_service.cc:1467] Processing CreateTablet for tablet ea6e4a74e7b44f99888206617865cee8 (DEFAULT_TABLE table=test-table [id=d655af10299a45859a629073fa6ed292]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:04.228967 3524 tablet_service.cc:1467] Processing CreateTablet for tablet ea6e4a74e7b44f99888206617865cee8 (DEFAULT_TABLE table=test-table [id=d655af10299a45859a629073fa6ed292]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:04.229049 3663 tablet_service.cc:1467] Processing CreateTablet for tablet ea6e4a74e7b44f99888206617865cee8 (DEFAULT_TABLE table=test-table [id=d655af10299a45859a629073fa6ed292]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:04.229173 3804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6e4a74e7b44f99888206617865cee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:04.229364 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6e4a74e7b44f99888206617865cee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:04.229372 3663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6e4a74e7b44f99888206617865cee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:04.231303 3936 ts_tablet_manager.cc:1918] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:05:04.231453 3936 log.cc:1196] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/157b97251a84476cafa09501a91bea65
I20260620 19:05:04.231861 3936 ts_tablet_manager.cc:1939] T 157b97251a84476cafa09501a91bea65 P b83ba29e68cf43759bfb8117d6be5b8e: Deleting consensus metadata
I20260620 19:05:04.232313 3893 tablet_bootstrap.cc:492] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4: Bootstrap starting.
I20260620 19:05:04.232508 3895 tablet_bootstrap.cc:492] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e: Bootstrap starting.
I20260620 19:05:04.233512 3895 tablet_bootstrap.cc:654] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:04.233471 3373 catalog_manager.cc:4841] TS b83ba29e68cf43759bfb8117d6be5b8e (127.1.154.131:44421): tablet 157b97251a84476cafa09501a91bea65 (table test-table [id=7e3939f7749e42cca7c3b10d84631ec0]) successfully deleted
I20260620 19:05:04.233551 3893 tablet_bootstrap.cc:654] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:04.234763 3894 tablet_bootstrap.cc:492] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318: Bootstrap starting.
I20260620 19:05:04.235431 3895 tablet_bootstrap.cc:492] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e: No bootstrap required, opened a new log
I20260620 19:05:04.235512 3895 ts_tablet_manager.cc:1397] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 19:05:04.236038 3894 tablet_bootstrap.cc:654] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:04.235957 3895 raft_consensus.cc:348] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:04.236097 3895 raft_consensus.cc:374] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:04.236135 3895 raft_consensus.cc:729] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83ba29e68cf43759bfb8117d6be5b8e, State: Initialized, Role: FOLLOWER
I20260620 19:05:04.236228 3937 ts_tablet_manager.cc:1918] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:05:04.236256 3895 consensus_queue.cc:260] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:04.236340 3937 log.cc:1196] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/wal/wals/157b97251a84476cafa09501a91bea65
I20260620 19:05:04.236562 3895 ts_tablet_manager.cc:1428] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:05:04.236662 3937 ts_tablet_manager.cc:1939] T 157b97251a84476cafa09501a91bea65 P 7931adc7007e4c2eadc82116065512d4: Deleting consensus metadata
I20260620 19:05:04.238006 3373 catalog_manager.cc:4841] TS 7931adc7007e4c2eadc82116065512d4 (127.1.154.129:39611): tablet 157b97251a84476cafa09501a91bea65 (table test-table [id=7e3939f7749e42cca7c3b10d84631ec0]) successfully deleted
I20260620 19:05:04.238565 3893 tablet_bootstrap.cc:492] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4: No bootstrap required, opened a new log
I20260620 19:05:04.238678 3893 ts_tablet_manager.cc:1397] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 19:05:04.239362 3893 raft_consensus.cc:348] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:04.239576 3893 raft_consensus.cc:374] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:04.239802 3893 raft_consensus.cc:729] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7931adc7007e4c2eadc82116065512d4, State: Initialized, Role: FOLLOWER
I20260620 19:05:04.239995 3893 consensus_queue.cc:260] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:04.240353 3893 ts_tablet_manager.cc:1428] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 19:05:04.241959 3938 ts_tablet_manager.cc:1905] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:04.245112 3938 ts_tablet_manager.cc:1918] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:05:04.245208 3938 log.cc:1196] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/157b97251a84476cafa09501a91bea65
I20260620 19:05:04.245362 3894 tablet_bootstrap.cc:492] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318: No bootstrap required, opened a new log
I20260620 19:05:04.245440 3894 ts_tablet_manager.cc:1397] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260620 19:05:04.245528 3938 ts_tablet_manager.cc:1939] T 157b97251a84476cafa09501a91bea65 P 1f65966760694bbaa8167ccfaa2ae318: Deleting consensus metadata
I20260620 19:05:04.246016 3894 raft_consensus.cc:348] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:04.246152 3894 raft_consensus.cc:374] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:04.246188 3894 raft_consensus.cc:729] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f65966760694bbaa8167ccfaa2ae318, State: Initialized, Role: FOLLOWER
I20260620 19:05:04.246477 3894 consensus_queue.cc:260] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:04.246786 3371 catalog_manager.cc:4841] TS 1f65966760694bbaa8167ccfaa2ae318 (127.1.154.130:46493): tablet 157b97251a84476cafa09501a91bea65 (table test-table [id=7e3939f7749e42cca7c3b10d84631ec0]) successfully deleted
I20260620 19:05:04.247282 3894 ts_tablet_manager.cc:1428] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:05:04.399637 3901 raft_consensus.cc:482] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:04.399901 3901 raft_consensus.cc:504] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:04.401865 3901 leader_election.cc:288] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7931adc7007e4c2eadc82116065512d4 (127.1.154.129:39611), 1f65966760694bbaa8167ccfaa2ae318 (127.1.154.130:46493)
I20260620 19:05:04.408182 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:04.402691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:38683 (local address 127.1.154.129:39611)
0620 19:05:04.403264 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:04.403284 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:04.403304 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:05:04.403376 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:04.403379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:04.403424 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:04.403534 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:04.404037 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.405187 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.406650 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.407363 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.407538 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:04.407602 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:04.407877 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 19:05:04.407989 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":134,"threads_started":1}
I20260620 19:05:04.408193 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:04.402399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:39611 (local address 127.1.154.131:38683)
0620 19:05:04.402822 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:04.402841 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:04.402993 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:04.403549 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:04.403552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:04.403570 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:04.403851 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:04.403867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.405349 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.405354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:04.406480 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:04.406494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.407543 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.407547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:04.407668 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:04.407736 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 19:05:04.407836 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 19:05:04.407943 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":72,"thread_start_us":100,"threads_started":1}
I20260620 19:05:04.408260 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:04.402925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:36949 (local address 127.1.154.130:46493)
0620 19:05:04.403148 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:04.403154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:04.403215 (+ 61us) server_negotiation.cc:407] Connection header received
0620 19:05:04.403369 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:04.403372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:04.403431 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:04.403535 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:04.404310 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.405072 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.406823 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.407429 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.407546 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:04.407602 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:04.408038 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 19:05:04.408115 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:05:04.408813 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:04.402399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:46493 (local address 127.1.154.131:36949)
0620 19:05:04.403124 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:04.403138 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:04.403244 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:04.403671 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:04.403674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:04.403694 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:04.404026 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:04.404034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.405244 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.405248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:04.406655 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:04.406669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.407689 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.407692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:04.407861 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:04.407898 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 19:05:04.408490 (+ 592us) client_negotiation.cc:240] Negotiation successful
0620 19:05:04.408614 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":228,"thread_start_us":80,"threads_started":1}
I20260620 19:05:04.408931 3543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6e4a74e7b44f99888206617865cee8" candidate_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7931adc7007e4c2eadc82116065512d4" is_pre_election: true
I20260620 19:05:04.409142 3543 raft_consensus.cc:2454] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b83ba29e68cf43759bfb8117d6be5b8e in term 0.
I20260620 19:05:04.409628 3757 leader_election.cc:302] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7931adc7007e4c2eadc82116065512d4, b83ba29e68cf43759bfb8117d6be5b8e; no voters:
I20260620 19:05:04.409935 3684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6e4a74e7b44f99888206617865cee8" candidate_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f65966760694bbaa8167ccfaa2ae318" is_pre_election: true
I20260620 19:05:04.410290 3684 raft_consensus.cc:2454] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b83ba29e68cf43759bfb8117d6be5b8e in term 0.
I20260620 19:05:04.410686 3901 raft_consensus.cc:2781] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:04.410758 3901 raft_consensus.cc:482] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:04.410796 3901 raft_consensus.cc:3037] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:04.412523 3901 raft_consensus.cc:504] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:04.413097 3901 leader_election.cc:288] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [CANDIDATE]: Term 1 election: Requested vote from peers 7931adc7007e4c2eadc82116065512d4 (127.1.154.129:39611), 1f65966760694bbaa8167ccfaa2ae318 (127.1.154.130:46493)
I20260620 19:05:04.413784 3543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6e4a74e7b44f99888206617865cee8" candidate_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7931adc7007e4c2eadc82116065512d4"
I20260620 19:05:04.413800 3684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6e4a74e7b44f99888206617865cee8" candidate_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f65966760694bbaa8167ccfaa2ae318"
I20260620 19:05:04.413972 3684 raft_consensus.cc:3037] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:04.414036 3543 raft_consensus.cc:3037] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:04.415486 3684 raft_consensus.cc:2454] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83ba29e68cf43759bfb8117d6be5b8e in term 1.
I20260620 19:05:04.415812 3543 raft_consensus.cc:2454] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83ba29e68cf43759bfb8117d6be5b8e in term 1.
I20260620 19:05:04.415907 3755 leader_election.cc:302] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f65966760694bbaa8167ccfaa2ae318, b83ba29e68cf43759bfb8117d6be5b8e; no voters:
I20260620 19:05:04.416154 3901 raft_consensus.cc:2781] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:04.416427 3901 raft_consensus.cc:686] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [term 1 LEADER]: Becoming Leader. State: Replica: b83ba29e68cf43759bfb8117d6be5b8e, State: Running, Role: LEADER
I20260620 19:05:04.416726 3901 consensus_queue.cc:237] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:04.419759 3385 catalog_manager.cc:5495] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e reported cstate change: term changed from 0 to 1, leader changed from <none> to b83ba29e68cf43759bfb8117d6be5b8e (127.1.154.131). New cstate: current_term: 1 leader_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } health_report { overall_health: HEALTHY } } }
I20260620 19:05:04.442548 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:04.438354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.1.154.131:44421)
0620 19:05:04.438709 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:04.438716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:04.438727 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:05:04.438793 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:04.438795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:04.438848 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:04.438981 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:04.439327 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.440038 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.440918 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.441160 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.441273 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:04.441554 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:04.442101 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:04.442228 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:04.442291 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 19:05:04.442353 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":140,"threads_started":1}
I20260620 19:05:04.457170 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:04.452872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:46493 (local address 127.1.154.131:32927)
0620 19:05:04.453188 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:04.453203 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:04.453290 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:04.453694 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:04.453702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:04.453710 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:05:04.453829 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:04.453833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.454821 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.454823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:04.455498 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:04.455506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.456136 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.456141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:04.456306 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:04.456383 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:05:04.456762 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:05:04.456772 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 19:05:04.456861 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 19:05:04.456972 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":133,"threads_started":1}
I20260620 19:05:04.457152 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:04.453123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:32927 (local address 127.1.154.130:46493)
0620 19:05:04.453259 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:04.453263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:04.453281 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:05:04.453403 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:04.453405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:04.453461 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:04.453557 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:04.453995 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.454668 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.455654 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.455932 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.456147 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:04.456564 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:04.456663 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:04.456753 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:04.456883 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 19:05:04.456946 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:05:04.468887 3543 raft_consensus.cc:1261] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Refusing update from remote peer b83ba29e68cf43759bfb8117d6be5b8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:04.469727 3684 raft_consensus.cc:1261] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Refusing update from remote peer b83ba29e68cf43759bfb8117d6be5b8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:04.469661 3901 consensus_queue.cc:1035] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:04.471846 3901 consensus_queue.cc:1035] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:04.492029 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:04.486957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:44421 (local address 127.1.154.130:39835)
0620 19:05:04.487449 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:04.487468 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:04.487605 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:04.487929 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:04.487931 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:04.487944 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:05:04.488047 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:04.488052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.488937 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.488941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:04.490329 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:04.490338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.490902 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.490907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:04.491103 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:04.491143 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:05:04.491487 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:05:04.491494 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 19:05:04.491762 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 19:05:04.491871 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380}
I20260620 19:05:04.492034 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:04.487210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:39835 (local address 127.1.154.131:44421)
0620 19:05:04.487355 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:04.487359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:04.487556 (+ 197us) server_negotiation.cc:407] Connection header received
0620 19:05:04.487737 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:04.487742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:04.487815 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:04.487901 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:04.488147 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.488797 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.490484 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:04.490787 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:04.490880 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:04.491311 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:04.491416 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:04.491476 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:04.491768 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 19:05:04.491866 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 19:05:04.492883 3919 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 19:05:06.082504 3386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:06.082839 3386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:06.087157 3386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c: Sending DeleteTablet for 3 replicas of tablet ea6e4a74e7b44f99888206617865cee8
I20260620 19:05:06.087896 3663 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6e4a74e7b44f99888206617865cee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:06 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:33328
I20260620 19:05:06.088066 3524 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6e4a74e7b44f99888206617865cee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:06 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:45140
I20260620 19:05:06.088348 3976 tablet_replica.cc:330] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318: stopping tablet replica
I20260620 19:05:06.088384 3977 tablet_replica.cc:330] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4: stopping tablet replica
I20260620 19:05:06.088488 3977 raft_consensus.cc:2229] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:06.088488 3976 raft_consensus.cc:2229] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:06.088327 3804 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6e4a74e7b44f99888206617865cee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:06 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:44754
I20260620 19:05:06.088609 3976 raft_consensus.cc:2258] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:06.088609 3977 raft_consensus.cc:2258] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:06.089429 3976 ts_tablet_manager.cc:1905] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:06.089460 3978 tablet_replica.cc:330] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e: stopping tablet replica
I20260620 19:05:06.089578 3978 raft_consensus.cc:2229] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:05:06.090016 3978 raft_consensus.cc:2258] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:06.091091 3977 ts_tablet_manager.cc:1905] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:06.090691 3978 ts_tablet_manager.cc:1905] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:06.093349 3976 ts_tablet_manager.cc:1918] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:05:06.093443 3976 log.cc:1196] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/ea6e4a74e7b44f99888206617865cee8
I20260620 19:05:06.093772 3976 ts_tablet_manager.cc:1939] T ea6e4a74e7b44f99888206617865cee8 P 1f65966760694bbaa8167ccfaa2ae318: Deleting consensus metadata
I20260620 19:05:06.094316 3978 ts_tablet_manager.cc:1918] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:05:06.094412 3978 log.cc:1196] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/ea6e4a74e7b44f99888206617865cee8
I20260620 19:05:06.094609 3371 catalog_manager.cc:4841] TS 1f65966760694bbaa8167ccfaa2ae318 (127.1.154.130:46493): tablet ea6e4a74e7b44f99888206617865cee8 (table test-table [id=d655af10299a45859a629073fa6ed292]) successfully deleted
I20260620 19:05:06.094732 3978 ts_tablet_manager.cc:1939] T ea6e4a74e7b44f99888206617865cee8 P b83ba29e68cf43759bfb8117d6be5b8e: Deleting consensus metadata
I20260620 19:05:06.094744 3977 ts_tablet_manager.cc:1918] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:05:06.094823 3977 log.cc:1196] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/wal/wals/ea6e4a74e7b44f99888206617865cee8
I20260620 19:05:06.095474 3977 ts_tablet_manager.cc:1939] T ea6e4a74e7b44f99888206617865cee8 P 7931adc7007e4c2eadc82116065512d4: Deleting consensus metadata
I20260620 19:05:06.095491 3373 catalog_manager.cc:4841] TS b83ba29e68cf43759bfb8117d6be5b8e (127.1.154.131:44421): tablet ea6e4a74e7b44f99888206617865cee8 (table test-table [id=d655af10299a45859a629073fa6ed292]) successfully deleted
I20260620 19:05:06.097218 3373 catalog_manager.cc:4841] TS 7931adc7007e4c2eadc82116065512d4 (127.1.154.129:39611): tablet ea6e4a74e7b44f99888206617865cee8 (table test-table [id=d655af10299a45859a629073fa6ed292]) successfully deleted
Jun 20 19:05:06 dist-test-slave-6wrf krb5kdc[3342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982306, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:05:06 dist-test-slave-6wrf krb5kdc[3342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982306, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:06.137542 3985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:06.116114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58078 (local address 127.1.154.190:35543)
0620 19:05:06.116410 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:06.116417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:06.116746 (+ 329us) server_negotiation.cc:407] Connection header received
0620 19:05:06.116868 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:06.116871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:06.116932 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:06.117022 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:06.118024 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:06.118882 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:06.120710 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:06.121021 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:06.121379 (+ 358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:06.132392 (+ 11013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:06.132396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:06.132398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:06.132433 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:06.134990 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:06.135806 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:06.135811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:06.135816 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:06.135922 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:06.136300 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:06.136303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:06.136307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:06.136591 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:06.136746 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:06.137204 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 19:05:06.137373 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":132,"threads_started":1}
I20260620 19:05:06.142624 3386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:05:06.143298 3386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:06.150770 3524 tablet_service.cc:1467] Processing CreateTablet for tablet dbbdd008764242d1bc3f7e78a8dd2434 (DEFAULT_TABLE table=test-table [id=14d33d73fcbf4e989795fdf26dcf56e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:06.150952 3663 tablet_service.cc:1467] Processing CreateTablet for tablet dbbdd008764242d1bc3f7e78a8dd2434 (DEFAULT_TABLE table=test-table [id=14d33d73fcbf4e989795fdf26dcf56e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:06.151109 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbdd008764242d1bc3f7e78a8dd2434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:06.151207 3663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbdd008764242d1bc3f7e78a8dd2434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:06.151198 3804 tablet_service.cc:1467] Processing CreateTablet for tablet dbbdd008764242d1bc3f7e78a8dd2434 (DEFAULT_TABLE table=test-table [id=14d33d73fcbf4e989795fdf26dcf56e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:06.151648 3804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbdd008764242d1bc3f7e78a8dd2434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:06.154381 3987 tablet_bootstrap.cc:492] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4: Bootstrap starting.
I20260620 19:05:06.154780 3988 tablet_bootstrap.cc:492] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318: Bootstrap starting.
I20260620 19:05:06.154786 3989 tablet_bootstrap.cc:492] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e: Bootstrap starting.
I20260620 19:05:06.155751 3987 tablet_bootstrap.cc:654] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:06.156324 3988 tablet_bootstrap.cc:654] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:06.156324 3989 tablet_bootstrap.cc:654] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:06.157691 3989 tablet_bootstrap.cc:492] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e: No bootstrap required, opened a new log
I20260620 19:05:06.157691 3988 tablet_bootstrap.cc:492] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318: No bootstrap required, opened a new log
I20260620 19:05:06.157796 3989 ts_tablet_manager.cc:1397] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:06.157796 3988 ts_tablet_manager.cc:1397] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:06.158255 3989 raft_consensus.cc:348] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.158255 3988 raft_consensus.cc:348] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.158394 3988 raft_consensus.cc:374] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:06.158394 3989 raft_consensus.cc:374] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:06.158434 3988 raft_consensus.cc:729] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f65966760694bbaa8167ccfaa2ae318, State: Initialized, Role: FOLLOWER
I20260620 19:05:06.158434 3989 raft_consensus.cc:729] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83ba29e68cf43759bfb8117d6be5b8e, State: Initialized, Role: FOLLOWER
I20260620 19:05:06.158571 3989 consensus_queue.cc:260] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.158571 3988 consensus_queue.cc:260] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.158854 3989 ts_tablet_manager.cc:1428] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:05:06.158872 3988 ts_tablet_manager.cc:1428] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:05:06.160094 3987 tablet_bootstrap.cc:492] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4: No bootstrap required, opened a new log
I20260620 19:05:06.160248 3987 ts_tablet_manager.cc:1397] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260620 19:05:06.160892 3987 raft_consensus.cc:348] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.161566 3987 raft_consensus.cc:374] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:06.161656 3987 raft_consensus.cc:729] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7931adc7007e4c2eadc82116065512d4, State: Initialized, Role: FOLLOWER
I20260620 19:05:06.161839 3987 consensus_queue.cc:260] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.162431 3987 ts_tablet_manager.cc:1428] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:05:06.218608 3953 raft_consensus.cc:482] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:06.219071 3953 raft_consensus.cc:504] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.219693 3953 leader_election.cc:288] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7931adc7007e4c2eadc82116065512d4 (127.1.154.129:39611), 1f65966760694bbaa8167ccfaa2ae318 (127.1.154.130:46493)
I20260620 19:05:06.220269 3684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbdd008764242d1bc3f7e78a8dd2434" candidate_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f65966760694bbaa8167ccfaa2ae318" is_pre_election: true
I20260620 19:05:06.220453 3684 raft_consensus.cc:2454] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b83ba29e68cf43759bfb8117d6be5b8e in term 0.
I20260620 19:05:06.220382 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbdd008764242d1bc3f7e78a8dd2434" candidate_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7931adc7007e4c2eadc82116065512d4" is_pre_election: true
I20260620 19:05:06.220611 3544 raft_consensus.cc:2454] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b83ba29e68cf43759bfb8117d6be5b8e in term 0.
I20260620 19:05:06.220927 3755 leader_election.cc:302] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f65966760694bbaa8167ccfaa2ae318, b83ba29e68cf43759bfb8117d6be5b8e; no voters:
I20260620 19:05:06.221289 3953 raft_consensus.cc:2781] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:06.221364 3953 raft_consensus.cc:482] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:06.221398 3953 raft_consensus.cc:3037] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:06.222654 3953 raft_consensus.cc:504] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.223022 3953 leader_election.cc:288] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [CANDIDATE]: Term 1 election: Requested vote from peers 7931adc7007e4c2eadc82116065512d4 (127.1.154.129:39611), 1f65966760694bbaa8167ccfaa2ae318 (127.1.154.130:46493)
I20260620 19:05:06.223455 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbdd008764242d1bc3f7e78a8dd2434" candidate_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7931adc7007e4c2eadc82116065512d4"
I20260620 19:05:06.223455 3684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbdd008764242d1bc3f7e78a8dd2434" candidate_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f65966760694bbaa8167ccfaa2ae318"
I20260620 19:05:06.223608 3544 raft_consensus.cc:3037] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:06.223596 3684 raft_consensus.cc:3037] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:06.224889 3544 raft_consensus.cc:2454] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83ba29e68cf43759bfb8117d6be5b8e in term 1.
I20260620 19:05:06.224889 3684 raft_consensus.cc:2454] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83ba29e68cf43759bfb8117d6be5b8e in term 1.
I20260620 19:05:06.225333 3757 leader_election.cc:302] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7931adc7007e4c2eadc82116065512d4, b83ba29e68cf43759bfb8117d6be5b8e; no voters:
I20260620 19:05:06.225713 3953 raft_consensus.cc:2781] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:06.225819 3953 raft_consensus.cc:686] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [term 1 LEADER]: Becoming Leader. State: Replica: b83ba29e68cf43759bfb8117d6be5b8e, State: Running, Role: LEADER
I20260620 19:05:06.225950 3953 consensus_queue.cc:237] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.228960 3382 catalog_manager.cc:5495] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e reported cstate change: term changed from 0 to 1, leader changed from <none> to b83ba29e68cf43759bfb8117d6be5b8e (127.1.154.131). New cstate: current_term: 1 leader_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } health_report { overall_health: HEALTHY } } }
I20260620 19:05:06.256734 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:06.252228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44628 (local address 127.1.154.131:44421)
0620 19:05:06.252592 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:06.252600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:06.252613 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:05:06.252665 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:06.252668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:06.252710 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:06.252921 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:06.253205 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:06.253780 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:06.255281 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:06.255550 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:06.255669 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:06.256144 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:06.256253 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:06.256343 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:06.256494 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 19:05:06.256553 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":131,"threads_started":1}
I20260620 19:05:06.267135 3544 raft_consensus.cc:1261] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Refusing update from remote peer b83ba29e68cf43759bfb8117d6be5b8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:06.267170 3684 raft_consensus.cc:1261] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Refusing update from remote peer b83ba29e68cf43759bfb8117d6be5b8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:06.268090 3969 consensus_queue.cc:1035] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:06.268512 3953 consensus_queue.cc:1035] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:06.311905 3386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:06.312116 3386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:06.314800 3386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c: Sending DeleteTablet for 3 replicas of tablet dbbdd008764242d1bc3f7e78a8dd2434
I20260620 19:05:06.315584 3804 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbdd008764242d1bc3f7e78a8dd2434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:06 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:44754
I20260620 19:05:06.315670 3663 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbdd008764242d1bc3f7e78a8dd2434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:06 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:33328
I20260620 19:05:06.315831 3978 tablet_replica.cc:330] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e: stopping tablet replica
I20260620 19:05:06.315811 3524 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbdd008764242d1bc3f7e78a8dd2434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:06 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:45140
I20260620 19:05:06.315881 3976 tablet_replica.cc:330] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318: stopping tablet replica
I20260620 19:05:06.316026 3978 raft_consensus.cc:2229] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:05:06.316026 3976 raft_consensus.cc:2229] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:06.316053 3977 tablet_replica.cc:330] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4: stopping tablet replica
I20260620 19:05:06.316250 3978 raft_consensus.cc:2258] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:06.316331 3976 raft_consensus.cc:2258] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:06.316337 3977 raft_consensus.cc:2229] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:06.316465 3977 raft_consensus.cc:2258] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:06.317488 3976 ts_tablet_manager.cc:1905] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:06.317489 3977 ts_tablet_manager.cc:1905] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:06.318120 3978 ts_tablet_manager.cc:1905] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:06.320828 3978 ts_tablet_manager.cc:1918] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:05:06.320904 3978 log.cc:1196] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/dbbdd008764242d1bc3f7e78a8dd2434
I20260620 19:05:06.321135 3978 ts_tablet_manager.cc:1939] T dbbdd008764242d1bc3f7e78a8dd2434 P b83ba29e68cf43759bfb8117d6be5b8e: Deleting consensus metadata
I20260620 19:05:06.321566 3386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:05:06.322378 3386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:06.322430 3373 catalog_manager.cc:4841] TS b83ba29e68cf43759bfb8117d6be5b8e (127.1.154.131:44421): tablet dbbdd008764242d1bc3f7e78a8dd2434 (table test-table [id=14d33d73fcbf4e989795fdf26dcf56e5]) successfully deleted
I20260620 19:05:06.322444 3976 ts_tablet_manager.cc:1918] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:05:06.322727 3976 log.cc:1196] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/dbbdd008764242d1bc3f7e78a8dd2434
I20260620 19:05:06.323028 3976 ts_tablet_manager.cc:1939] T dbbdd008764242d1bc3f7e78a8dd2434 P 1f65966760694bbaa8167ccfaa2ae318: Deleting consensus metadata
I20260620 19:05:06.324368 3371 catalog_manager.cc:4841] TS 1f65966760694bbaa8167ccfaa2ae318 (127.1.154.130:46493): tablet dbbdd008764242d1bc3f7e78a8dd2434 (table test-table [id=14d33d73fcbf4e989795fdf26dcf56e5]) successfully deleted
I20260620 19:05:06.324419 3977 ts_tablet_manager.cc:1918] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:05:06.324564 3977 log.cc:1196] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-0/wal/wals/dbbdd008764242d1bc3f7e78a8dd2434
I20260620 19:05:06.324849 3977 ts_tablet_manager.cc:1939] T dbbdd008764242d1bc3f7e78a8dd2434 P 7931adc7007e4c2eadc82116065512d4: Deleting consensus metadata
I20260620 19:05:06.326345 3373 catalog_manager.cc:4841] TS 7931adc7007e4c2eadc82116065512d4 (127.1.154.129:39611): tablet dbbdd008764242d1bc3f7e78a8dd2434 (table test-table [id=14d33d73fcbf4e989795fdf26dcf56e5]) successfully deleted
I20260620 19:05:06.330169 3524 tablet_service.cc:1467] Processing CreateTablet for tablet fcf2f1e6043b4c1991ef5611fc95e72a (DEFAULT_TABLE table=test-table [id=388eff293a404596ac4ebe85514bf34e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:06.330545 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcf2f1e6043b4c1991ef5611fc95e72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:06.330719 3804 tablet_service.cc:1467] Processing CreateTablet for tablet fcf2f1e6043b4c1991ef5611fc95e72a (DEFAULT_TABLE table=test-table [id=388eff293a404596ac4ebe85514bf34e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:06.330749 3663 tablet_service.cc:1467] Processing CreateTablet for tablet fcf2f1e6043b4c1991ef5611fc95e72a (DEFAULT_TABLE table=test-table [id=388eff293a404596ac4ebe85514bf34e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:06.331012 3804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcf2f1e6043b4c1991ef5611fc95e72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:06.331153 3663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcf2f1e6043b4c1991ef5611fc95e72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:06.333503 3988 tablet_bootstrap.cc:492] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318: Bootstrap starting.
I20260620 19:05:06.333950 3989 tablet_bootstrap.cc:492] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e: Bootstrap starting.
I20260620 19:05:06.334477 3988 tablet_bootstrap.cc:654] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:06.335011 3987 tablet_bootstrap.cc:492] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4: Bootstrap starting.
I20260620 19:05:06.335292 3989 tablet_bootstrap.cc:654] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:06.335858 3988 tablet_bootstrap.cc:492] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318: No bootstrap required, opened a new log
I20260620 19:05:06.335935 3988 ts_tablet_manager.cc:1397] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:05:06.335807 3987 tablet_bootstrap.cc:654] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:06.336524 3988 raft_consensus.cc:348] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.336696 3989 tablet_bootstrap.cc:492] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e: No bootstrap required, opened a new log
I20260620 19:05:06.336709 3988 raft_consensus.cc:374] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:06.336755 3988 raft_consensus.cc:729] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f65966760694bbaa8167ccfaa2ae318, State: Initialized, Role: FOLLOWER
I20260620 19:05:06.336783 3989 ts_tablet_manager.cc:1397] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:06.336926 3988 consensus_queue.cc:260] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.337198 3988 ts_tablet_manager.cc:1428] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 19:05:06.337181 3989 raft_consensus.cc:348] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.337734 3989 raft_consensus.cc:374] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:06.337795 3989 raft_consensus.cc:729] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83ba29e68cf43759bfb8117d6be5b8e, State: Initialized, Role: FOLLOWER
I20260620 19:05:06.337951 3989 consensus_queue.cc:260] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.338030 3987 tablet_bootstrap.cc:492] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4: No bootstrap required, opened a new log
I20260620 19:05:06.338135 3987 ts_tablet_manager.cc:1397] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 19:05:06.338236 3989 ts_tablet_manager.cc:1428] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:05:06.338599 3987 raft_consensus.cc:348] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.338718 3987 raft_consensus.cc:374] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:06.338758 3987 raft_consensus.cc:729] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7931adc7007e4c2eadc82116065512d4, State: Initialized, Role: FOLLOWER
I20260620 19:05:06.338878 3987 consensus_queue.cc:260] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.339174 3987 ts_tablet_manager.cc:1428] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:05:06.647861 3971 raft_consensus.cc:482] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:06.648031 3971 raft_consensus.cc:504] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.648520 3971 leader_election.cc:288] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7931adc7007e4c2eadc82116065512d4 (127.1.154.129:39611), b83ba29e68cf43759bfb8117d6be5b8e (127.1.154.131:44421)
I20260620 19:05:06.648732 3993 raft_consensus.cc:482] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:06.648988 3993 raft_consensus.cc:504] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.649559 3543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf2f1e6043b4c1991ef5611fc95e72a" candidate_uuid: "1f65966760694bbaa8167ccfaa2ae318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7931adc7007e4c2eadc82116065512d4" is_pre_election: true
I20260620 19:05:06.649518 3823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf2f1e6043b4c1991ef5611fc95e72a" candidate_uuid: "1f65966760694bbaa8167ccfaa2ae318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" is_pre_election: true
I20260620 19:05:06.649699 3543 raft_consensus.cc:2454] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f65966760694bbaa8167ccfaa2ae318 in term 0.
I20260620 19:05:06.649726 3823 raft_consensus.cc:2454] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f65966760694bbaa8167ccfaa2ae318 in term 0.
I20260620 19:05:06.650159 3617 leader_election.cc:302] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f65966760694bbaa8167ccfaa2ae318, b83ba29e68cf43759bfb8117d6be5b8e; no voters:
I20260620 19:05:06.650511 3971 raft_consensus.cc:2781] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:06.650580 3971 raft_consensus.cc:482] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:06.650591 3993 leader_election.cc:288] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f65966760694bbaa8167ccfaa2ae318 (127.1.154.130:46493), b83ba29e68cf43759bfb8117d6be5b8e (127.1.154.131:44421)
I20260620 19:05:06.650635 3971 raft_consensus.cc:3037] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:06.652460 3971 raft_consensus.cc:504] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.652781 3971 leader_election.cc:288] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [CANDIDATE]: Term 1 election: Requested vote from peers 7931adc7007e4c2eadc82116065512d4 (127.1.154.129:39611), b83ba29e68cf43759bfb8117d6be5b8e (127.1.154.131:44421)
I20260620 19:05:06.653133 3823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf2f1e6043b4c1991ef5611fc95e72a" candidate_uuid: "1f65966760694bbaa8167ccfaa2ae318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83ba29e68cf43759bfb8117d6be5b8e"
I20260620 19:05:06.653133 3543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf2f1e6043b4c1991ef5611fc95e72a" candidate_uuid: "1f65966760694bbaa8167ccfaa2ae318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7931adc7007e4c2eadc82116065512d4"
I20260620 19:05:06.653294 3823 raft_consensus.cc:3037] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:06.653251 3543 raft_consensus.cc:3037] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:06.654448 3543 raft_consensus.cc:2454] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f65966760694bbaa8167ccfaa2ae318 in term 1.
I20260620 19:05:06.654649 3823 raft_consensus.cc:2454] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f65966760694bbaa8167ccfaa2ae318 in term 1.
I20260620 19:05:06.654767 3617 leader_election.cc:302] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f65966760694bbaa8167ccfaa2ae318, 7931adc7007e4c2eadc82116065512d4; no voters:
I20260620 19:05:06.655123 3971 raft_consensus.cc:2781] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:06.655223 3971 raft_consensus.cc:686] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [term 1 LEADER]: Becoming Leader. State: Replica: 1f65966760694bbaa8167ccfaa2ae318, State: Running, Role: LEADER
I20260620 19:05:06.655352 3971 consensus_queue.cc:237] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } }
I20260620 19:05:06.656539 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:06.651532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:39063 (local address 127.1.154.131:44421)
0620 19:05:06.651663 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:06.651667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:06.651679 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:05:06.651720 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:06.651723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:06.651765 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:06.651884 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:06.652137 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:06.652792 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:06.655111 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:06.655720 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:06.655876 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:06.655936 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:06.656320 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 19:05:06.656395 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:05:06.656916 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:06.650899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:44421 (local address 127.1.154.129:39063)
0620 19:05:06.651273 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:06.651286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:06.651402 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:06.651875 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:06.651879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:06.651895 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:06.652016 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:06.652023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:06.652939 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:06.652945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:06.654943 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:06.654949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:06.656018 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:06.656024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:06.656157 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:06.656183 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 19:05:06.656677 (+ 494us) client_negotiation.cc:240] Negotiation successful
0620 19:05:06.656770 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"spinlock_wait_cycles":7424,"thread_start_us":128,"threads_started":1}
I20260620 19:05:06.657296 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:06.650899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:34825 (local address 127.1.154.130:46493)
0620 19:05:06.651143 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:06.651157 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:06.651319 (+ 162us) server_negotiation.cc:407] Connection header received
0620 19:05:06.651535 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:06.651538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:06.651588 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:06.651670 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:06.652015 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:06.652803 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:06.655221 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:06.656180 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:06.656382 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:06.656453 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:06.656994 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 19:05:06.657166 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":49,"threads_started":1}
I20260620 19:05:06.657270 3387 catalog_manager.cc:5495] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f65966760694bbaa8167ccfaa2ae318 (127.1.154.130). New cstate: current_term: 1 leader_uuid: "1f65966760694bbaa8167ccfaa2ae318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f65966760694bbaa8167ccfaa2ae318" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 } health_report { overall_health: UNKNOWN } } }
I20260620 19:05:06.657702 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:06.650769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:46493 (local address 127.1.154.129:34825)
0620 19:05:06.651212 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:06.651228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:06.651402 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:06.651708 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:06.651711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:06.651733 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:06.651875 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:06.651881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:06.652977 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:06.652980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:06.654894 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:06.654919 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:06.656547 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:06.656551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:06.656697 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:06.656730 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 19:05:06.657437 (+ 707us) client_negotiation.cc:240] Negotiation successful
0620 19:05:06.657530 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":34,"spinlock_wait_cycles":25856,"thread_start_us":89,"threads_started":1}
I20260620 19:05:06.658095 3684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf2f1e6043b4c1991ef5611fc95e72a" candidate_uuid: "7931adc7007e4c2eadc82116065512d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f65966760694bbaa8167ccfaa2ae318" is_pre_election: true
I20260620 19:05:06.658118 3823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf2f1e6043b4c1991ef5611fc95e72a" candidate_uuid: "7931adc7007e4c2eadc82116065512d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" is_pre_election: true
I20260620 19:05:06.658306 3823 raft_consensus.cc:2379] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7931adc7007e4c2eadc82116065512d4 in current term 1: Already voted for candidate 1f65966760694bbaa8167ccfaa2ae318 in this term.
I20260620 19:05:06.658676 3477 leader_election.cc:302] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7931adc7007e4c2eadc82116065512d4; no voters: 1f65966760694bbaa8167ccfaa2ae318, b83ba29e68cf43759bfb8117d6be5b8e
I20260620 19:05:06.658979 3993 raft_consensus.cc:2726] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:05:06.724620 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:06.721092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.1.154.130:46493)
0620 19:05:06.721277 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:06.721283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:06.721301 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:05:06.721398 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:06.721400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:06.721441 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:06.721502 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:06.721802 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:06.722319 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:06.723235 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:06.723704 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:06.723798 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:06.724080 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:06.724170 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:06.724239 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:06.724337 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 19:05:06.724438 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 19:05:06.733342 3823 raft_consensus.cc:1261] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Refusing update from remote peer 1f65966760694bbaa8167ccfaa2ae318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:06.733345 3543 raft_consensus.cc:1261] T fcf2f1e6043b4c1991ef5611fc95e72a P 7931adc7007e4c2eadc82116065512d4 [term 1 FOLLOWER]: Refusing update from remote peer 1f65966760694bbaa8167ccfaa2ae318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:06.733863 3900 consensus_queue.cc:1035] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7931adc7007e4c2eadc82116065512d4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 39611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:06.734084 3995 consensus_queue.cc:1035] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83ba29e68cf43759bfb8117d6be5b8e" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:05:06.749577 3955 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 19:05:08.327546 3382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:08.327710 3382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:08.331530 3382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4be88a323bb44a3d8a01d78aa908736c: Sending DeleteTablet for 3 replicas of tablet fcf2f1e6043b4c1991ef5611fc95e72a
I20260620 19:05:08.332316 3663 tablet_service.cc:1514] Processing DeleteTablet for tablet fcf2f1e6043b4c1991ef5611fc95e72a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:08 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:33328
I20260620 19:05:08.332316 3524 tablet_service.cc:1514] Processing DeleteTablet for tablet fcf2f1e6043b4c1991ef5611fc95e72a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:08 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:45140
I20260620 19:05:08.332490 3804 tablet_service.cc:1514] Processing DeleteTablet for tablet fcf2f1e6043b4c1991ef5611fc95e72a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:08 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:44754
I20260620 19:05:08.332728 4035 tablet_replica.cc:330] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318: stopping tablet replica
I20260620 19:05:08.332872 4035 raft_consensus.cc:2229] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:05:08.332928 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 3459
I20260620 19:05:08.334493 4035 raft_consensus.cc:2258] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:08.334724 4036 tablet_replica.cc:330] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e: stopping tablet replica
I20260620 19:05:08.334990 4036 raft_consensus.cc:2229] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:08.335186 4036 raft_consensus.cc:2258] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:08.337353 4035 ts_tablet_manager.cc:1905] T fcf2f1e6043b4c1991ef5611fc95e72a P 1f65966760694bbaa8167ccfaa2ae318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:08.337393 4036 ts_tablet_manager.cc:1905] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:05:08.339958 3373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:39611 (error 108)
I20260620 19:05:08.340265 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 3599
I20260620 19:05:08.342671 4038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:08.342028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:39611 (local address 127.0.0.1:35610)
0620 19:05:08.342423 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:08.342529 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":174,"threads_started":1}
W20260620 19:05:08.342958 3373 catalog_manager.cc:4584] TS 7931adc7007e4c2eadc82116065512d4 (127.1.154.129:39611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fcf2f1e6043b4c1991ef5611fc95e72a: Network error: Client connection negotiation failed: client connection to 127.1.154.129:39611: connect: Connection refused (error 111)
I20260620 19:05:08.343029 4036 ts_tablet_manager.cc:1918] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:05:08.343122 4036 log.cc:1196] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/fcf2f1e6043b4c1991ef5611fc95e72a
I20260620 19:05:08.343422 4036 ts_tablet_manager.cc:1939] T fcf2f1e6043b4c1991ef5611fc95e72a P b83ba29e68cf43759bfb8117d6be5b8e: Deleting consensus metadata
I20260620 19:05:08.344300 3373 catalog_manager.cc:4841] TS b83ba29e68cf43759bfb8117d6be5b8e (127.1.154.131:44421): tablet fcf2f1e6043b4c1991ef5611fc95e72a (table test-table [id=388eff293a404596ac4ebe85514bf34e]) successfully deleted
I20260620 19:05:08.348556 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 3739
I20260620 19:05:08.350270 4038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:08.349014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:46493 (local address 127.0.0.1:45066)
0620 19:05:08.350025 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:08.350127 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.130:46493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":863}
I20260620 19:05:08.356398 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 3355
2026-06-20T19:05:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7097 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:05:08 dist-test-slave-6wrf krb5kdc[4041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:05:08 dist-test-slave-6wrf krb5kdc[4041](info): set up 2 sockets
Jun 20 19:05:08 dist-test-slave-6wrf krb5kdc[4041](info): commencing operation
krb5kdc: starting...
W20260620 19:05:09.453996 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:05:09 dist-test-slave-6wrf krb5kdc[4041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:05:09.598465 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:46559
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:46559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:09.721313 4054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:09.721619 4054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:09.721662 4054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:09.725778 4054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:09.725842 4054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:09.725862 4054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:09.725879 4054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:09.725894 4054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:09.731128 4054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:46559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:46559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:09.732211 4054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:09.733235 4054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:09.742898 4062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:09.742880 4060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:09.743005 4054 server_base.cc:1034] running on GCE node
W20260620 19:05:09.742880 4059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:09.743759 4054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:09.744969 4054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:09.746433 4054 hybrid_clock.cc:648] HybridClock initialized: now 1781982309746371 us; error 92 us; skew 500 ppm
Jun 20 19:05:09 dist-test-slave-6wrf krb5kdc[4041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:09.750041 4054 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:05:09.751027 4054 webserver.cc:460] Webserver started at http://127.1.154.190:32787/ using document root <none> and password file <none>
I20260620 19:05:09.751609 4054 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:09.751701 4054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:09.751933 4054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:09.754103 4054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "3b2a7114a65d4e27869b840405d99a6f"
format_stamp: "Formatted at 2026-06-20 19:05:09 on dist-test-slave-6wrf"
server_key: "449d9519602146fb3e6907bad7381f27"
server_key_iv: "2dc73b36e1f1a8f311f6fba0237ba6b9"
server_key_version: "encryptionkey@0"
I20260620 19:05:09.754673 4054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "3b2a7114a65d4e27869b840405d99a6f"
format_stamp: "Formatted at 2026-06-20 19:05:09 on dist-test-slave-6wrf"
server_key: "449d9519602146fb3e6907bad7381f27"
server_key_iv: "2dc73b36e1f1a8f311f6fba0237ba6b9"
server_key_version: "encryptionkey@0"
I20260620 19:05:09.758877 4054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:05:09.761886 4069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:09.763468 4054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:09.763631 4054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "3b2a7114a65d4e27869b840405d99a6f"
format_stamp: "Formatted at 2026-06-20 19:05:09 on dist-test-slave-6wrf"
server_key: "449d9519602146fb3e6907bad7381f27"
server_key_iv: "2dc73b36e1f1a8f311f6fba0237ba6b9"
server_key_version: "encryptionkey@0"
I20260620 19:05:09.763753 4054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:09.792248 4054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:09.796761 4054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:09.796993 4054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:09.805071 4054 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:46559
I20260620 19:05:09.805085 4121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:46559 every 8 connection(s)
I20260620 19:05:09.806136 4054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:09.808120 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 4054
I20260620 19:05:09.808269 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:09.808571 1642 external_mini_cluster.cc:1428] Setting key 6eb7bf334a0b6cd114432d90fd12350d
I20260620 19:05:09.809626 4122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:09.817449 4122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f: Bootstrap starting.
Jun 20 19:05:09 dist-test-slave-6wrf krb5kdc[4041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:09.820135 4122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:09.820871 4122 log.cc:824] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:09.823590 4125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:09.810009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.1.154.190:46559)
0620 19:05:09.810624 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:09.810649 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:09.810710 (+ 61us) server_negotiation.cc:407] Connection header received
0620 19:05:09.811743 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:09.811766 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:09.812112 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:09.812562 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:09.813388 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:09.814495 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:09.815338 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:09.815816 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:09.816006 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:09.818985 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:09.818990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:09.819002 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:09.819029 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:09.820800 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:09.821264 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:09.821276 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:09.821281 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:09.821386 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:09.821714 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:09.821717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:09.821719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:09.822113 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:09.822322 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:09.822574 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 19:05:09.822857 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":145,"threads_started":1}
I20260620 19:05:09.823896 4122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f: No bootstrap required, opened a new log
I20260620 19:05:09.827493 4122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2a7114a65d4e27869b840405d99a6f" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46559 } }
I20260620 19:05:09.827759 4122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:09.827837 4122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b2a7114a65d4e27869b840405d99a6f, State: Initialized, Role: FOLLOWER
I20260620 19:05:09.828259 4122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2a7114a65d4e27869b840405d99a6f" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46559 } }
I20260620 19:05:09.828418 4122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:09.828502 4122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:09.828616 4122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:09.829571 4122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2a7114a65d4e27869b840405d99a6f" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46559 } }
I20260620 19:05:09.829988 4122 leader_election.cc:302] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b2a7114a65d4e27869b840405d99a6f; no voters:
I20260620 19:05:09.830317 4122 leader_election.cc:288] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:09.830405 4127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:09.830598 4127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [term 1 LEADER]: Becoming Leader. State: Replica: 3b2a7114a65d4e27869b840405d99a6f, State: Running, Role: LEADER
I20260620 19:05:09.831010 4127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2a7114a65d4e27869b840405d99a6f" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46559 } }
I20260620 19:05:09.831624 4122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:09.832616 4128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b2a7114a65d4e27869b840405d99a6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2a7114a65d4e27869b840405d99a6f" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46559 } } }
I20260620 19:05:09.832967 4129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b2a7114a65d4e27869b840405d99a6f. Latest consensus state: current_term: 1 leader_uuid: "3b2a7114a65d4e27869b840405d99a6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2a7114a65d4e27869b840405d99a6f" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46559 } } }
I20260620 19:05:09.833112 4128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:09.833812 4138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:09.833853 4129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:09.836483 4138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:09.842465 4138 catalog_manager.cc:1347] Generated new cluster ID: 4cf8010e2c214e5980c8efd8ba0c973d
I20260620 19:05:09.842569 4138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:09.856756 4138 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:09.858297 4138 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:09.869675 4138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b2a7114a65d4e27869b840405d99a6f: Generated new TSK 0
I20260620 19:05:09.870833 4138 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:05:09.913856 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46559
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:10.033876 4148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:10.034138 4148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:10.034180 4148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:10.038168 4148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:10.038229 4148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:10.038286 4148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:05:10.044023 4148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:10.045773 4148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:10.046868 4148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:10.056972 4153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:10.056996 4154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:10.057168 4156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:10.057319 4148 server_base.cc:1034] running on GCE node
I20260620 19:05:10.057822 4148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:10.058466 4148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:10.059629 4148 hybrid_clock.cc:648] HybridClock initialized: now 1781982310059610 us; error 32 us; skew 500 ppm
Jun 20 19:05:10 dist-test-slave-6wrf krb5kdc[4041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982310, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:10.063459 4148 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:05:10.064728 4148 webserver.cc:460] Webserver started at http://127.1.154.129:43249/ using document root <none> and password file <none>
I20260620 19:05:10.065292 4148 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:10.065353 4148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:10.065531 4148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:10.067672 4148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "425fc6db3450416886828cde5e7b0073"
format_stamp: "Formatted at 2026-06-20 19:05:10 on dist-test-slave-6wrf"
server_key: "32a45253563b8eb7af402c3006b61512"
server_key_iv: "a88a0d28d62d3dd819b2474ab77f0ec7"
server_key_version: "encryptionkey@0"
I20260620 19:05:10.068308 4148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "425fc6db3450416886828cde5e7b0073"
format_stamp: "Formatted at 2026-06-20 19:05:10 on dist-test-slave-6wrf"
server_key: "32a45253563b8eb7af402c3006b61512"
server_key_iv: "a88a0d28d62d3dd819b2474ab77f0ec7"
server_key_version: "encryptionkey@0"
I20260620 19:05:10.072356 4148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:05:10.075098 4163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:10.077121 4148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:05:10.077293 4148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "425fc6db3450416886828cde5e7b0073"
format_stamp: "Formatted at 2026-06-20 19:05:10 on dist-test-slave-6wrf"
server_key: "32a45253563b8eb7af402c3006b61512"
server_key_iv: "a88a0d28d62d3dd819b2474ab77f0ec7"
server_key_version: "encryptionkey@0"
I20260620 19:05:10.077420 4148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:10.098925 4148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:10.102578 4148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:10.102797 4148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:10.103439 4148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:10.104290 4148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:10.104362 4148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:10.104430 4148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:10.104487 4148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:10.114368 4148 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:43467
I20260620 19:05:10.114463 4276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:43467 every 8 connection(s)
I20260620 19:05:10.115368 4148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:05:10.120859 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 4148
I20260620 19:05:10.120993 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:05:10.121304 1642 external_mini_cluster.cc:1428] Setting key 188e78797c11a49d856a061a2c9c3f38
Jun 20 19:05:10 dist-test-slave-6wrf krb5kdc[4041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982310, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:10.130743 4125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:10.117663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:60505 (local address 127.1.154.190:46559)
0620 19:05:10.117828 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:10.117831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:10.118685 (+ 854us) server_negotiation.cc:407] Connection header received
0620 19:05:10.120001 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:10.120004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:10.120060 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:10.120142 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:10.121735 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:10.122443 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:10.124011 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:10.124243 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:10.124376 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:10.127023 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:10.127026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:10.127029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:10.127060 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:10.128621 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:10.129181 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:10.129185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:10.129187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:10.129264 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:10.129658 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:10.129661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:10.129663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:10.129844 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:10.129954 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:10.130440 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 19:05:10.130566 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 19:05:10.131587 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:10.118062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:46559 (local address 127.1.154.129:60505)
0620 19:05:10.118517 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:10.118559 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:10.119769 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:10.120313 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:10.120319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:10.120822 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:10.121494 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:10.121517 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:10.122575 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:10.122579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:10.123858 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:10.123872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:10.124445 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:10.124452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:10.124680 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:10.125321 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:10.125337 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:10.126864 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:10.128766 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:10.128772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:10.128787 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:10.129061 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:10.129356 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:10.129360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:10.129362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:10.129534 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:10.129951 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:10.129957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:10.130208 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 19:05:10.130390 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 19:05:10.130717 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
I20260620 19:05:10.133492 4277 heartbeater.cc:347] Connected to a master server at 127.1.154.190:46559
I20260620 19:05:10.133785 4277 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:10.134312 4277 heartbeater.cc:511] Master 127.1.154.190:46559 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
I20260620 19:05:10.136198 4080 ts_manager.cc:194] Registered new tserver with Master: 425fc6db3450416886828cde5e7b0073 (127.1.154.129:43467)
I20260620 19:05:10.137318 4080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:60505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:05:10.157007 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46559
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:10.272883 4282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:10.273159 4282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:10.273202 4282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:10.277356 4282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:10.277546 4282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:10.277773 4282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:10.285439 4282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:10.286720 4282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:10.287732 4282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:10.298983 4287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:10.298995 4290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:10.299440 4288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:10.300120 4282 server_base.cc:1034] running on GCE node
I20260620 19:05:10.300603 4282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:10.301285 4282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:10.302517 4282 hybrid_clock.cc:648] HybridClock initialized: now 1781982310302518 us; error 68 us; skew 500 ppm
Jun 20 19:05:10 dist-test-slave-6wrf krb5kdc[4041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982310, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:10.305325 4282 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:05:10.306387 4282 webserver.cc:460] Webserver started at http://127.1.154.130:38767/ using document root <none> and password file <none>
I20260620 19:05:10.306968 4282 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:10.307058 4282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:10.307284 4282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:10.309327 4282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "7ecc27b0e2d84ff2ba864025b303d747"
format_stamp: "Formatted at 2026-06-20 19:05:10 on dist-test-slave-6wrf"
server_key: "5c93be152ca3e606f321b68839afb294"
server_key_iv: "ad8588aa15446754213a05d88073a6d5"
server_key_version: "encryptionkey@0"
I20260620 19:05:10.309934 4282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "7ecc27b0e2d84ff2ba864025b303d747"
format_stamp: "Formatted at 2026-06-20 19:05:10 on dist-test-slave-6wrf"
server_key: "5c93be152ca3e606f321b68839afb294"
server_key_iv: "ad8588aa15446754213a05d88073a6d5"
server_key_version: "encryptionkey@0"
I20260620 19:05:10.315024 4282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 19:05:10.318555 4297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:10.320081 4282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:05:10.320267 4282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "7ecc27b0e2d84ff2ba864025b303d747"
format_stamp: "Formatted at 2026-06-20 19:05:10 on dist-test-slave-6wrf"
server_key: "5c93be152ca3e606f321b68839afb294"
server_key_iv: "ad8588aa15446754213a05d88073a6d5"
server_key_version: "encryptionkey@0"
I20260620 19:05:10.320422 4282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:10.375936 4282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:10.379418 4282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:10.379626 4282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:10.380230 4282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:10.381170 4282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:10.381243 4282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:10.381314 4282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:10.381364 4282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:10.391224 4282 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:45629
I20260620 19:05:10.391242 4410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:45629 every 8 connection(s)
I20260620 19:05:10.392127 4282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:10.395232 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 4282
I20260620 19:05:10.395339 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:05:10.395674 1642 external_mini_cluster.cc:1428] Setting key 76b9943f0689cc2cd90b9ca2138598be
Jun 20 19:05:10 dist-test-slave-6wrf krb5kdc[4041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982310, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
I20260620 19:05:10.406880 4125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:10.394072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:53803 (local address 127.1.154.190:46559)
0620 19:05:10.394244 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:10.394249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:10.395068 (+ 819us) server_negotiation.cc:407] Connection header received
0620 19:05:10.396655 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:10.396659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:10.396709 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:10.396789 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:10.398311 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:10.398890 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:10.400044 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:10.400285 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:10.400397 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:10.403266 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:10.403271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:10.403278 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:10.403302 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:10.404806 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:10.405305 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:10.405310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:10.405311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:10.405391 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:10.405800 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:10.405803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:10.405805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:10.405990 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:10.406095 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:10.406592 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 19:05:10.406712 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
I20260620 19:05:10.407788 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:10.394412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:46559 (local address 127.1.154.130:53803)
0620 19:05:10.394860 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:10.394900 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:10.396415 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:10.396947 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:10.396955 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:10.397418 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:10.398140 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:10.398162 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:10.399027 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:10.399033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:10.399883 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:10.399893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:10.400443 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:10.400447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:10.400617 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:10.401208 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:10.401224 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:10.403092 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:10.404924 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:10.404930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:10.404944 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:10.405198 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:10.405491 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:10.405496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:10.405498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:10.405697 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:10.406105 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:10.406113 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:10.406355 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 19:05:10.406538 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 19:05:10.406853 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":116,"threads_started":1}
I20260620 19:05:10.410435 4411 heartbeater.cc:347] Connected to a master server at 127.1.154.190:46559
I20260620 19:05:10.410734 4411 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:10.411252 4411 heartbeater.cc:511] Master 127.1.154.190:46559 requested a full tablet report, sending...
I20260620 19:05:10.412653 4080 ts_manager.cc:194] Registered new tserver with Master: 7ecc27b0e2d84ff2ba864025b303d747 (127.1.154.130:45629)
I20260620 19:05:10.413900 4080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:53803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:05:10.428359 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46559
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:10.549475 4416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:10.549901 4416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:10.549965 4416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:10.554111 4416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:10.554172 4416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:10.554235 4416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:10.559484 4416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:10.560990 4416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:10.562073 4416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:10.573132 4421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:10.573279 4416 server_base.cc:1034] running on GCE node
W20260620 19:05:10.573139 4422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:10.573158 4424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:10.574415 4416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:10.575134 4416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:10.578406 4416 hybrid_clock.cc:648] HybridClock initialized: now 1781982310578379 us; error 63 us; skew 500 ppm
Jun 20 19:05:10 dist-test-slave-6wrf krb5kdc[4041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982310, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:10.582683 4416 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:05:10.583685 4416 webserver.cc:460] Webserver started at http://127.1.154.131:33097/ using document root <none> and password file <none>
I20260620 19:05:10.584321 4416 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:10.584442 4416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:10.584667 4416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:10.586902 4416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "c567c7167acd464285db0da79512b9cd"
format_stamp: "Formatted at 2026-06-20 19:05:10 on dist-test-slave-6wrf"
server_key: "47674a59fe390deeac886a2e45eb8a83"
server_key_iv: "8ac220afe2e6a324a5917094e1a146a2"
server_key_version: "encryptionkey@0"
I20260620 19:05:10.587504 4416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "c567c7167acd464285db0da79512b9cd"
format_stamp: "Formatted at 2026-06-20 19:05:10 on dist-test-slave-6wrf"
server_key: "47674a59fe390deeac886a2e45eb8a83"
server_key_iv: "8ac220afe2e6a324a5917094e1a146a2"
server_key_version: "encryptionkey@0"
I20260620 19:05:10.592223 4416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 19:05:10.594995 4431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:10.596884 4416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:05:10.597054 4416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "c567c7167acd464285db0da79512b9cd"
format_stamp: "Formatted at 2026-06-20 19:05:10 on dist-test-slave-6wrf"
server_key: "47674a59fe390deeac886a2e45eb8a83"
server_key_iv: "8ac220afe2e6a324a5917094e1a146a2"
server_key_version: "encryptionkey@0"
I20260620 19:05:10.597345 4416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:10.622792 4416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:10.626783 4416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:10.626998 4416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:10.627610 4416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:10.628484 4416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:10.628530 4416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:10.628571 4416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:10.628587 4416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:10.640242 4416 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:35905
I20260620 19:05:10.640396 4544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:35905 every 8 connection(s)
I20260620 19:05:10.641278 4416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:10.646950 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 4416
I20260620 19:05:10.647228 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:05:10.647718 1642 external_mini_cluster.cc:1428] Setting key 6d4d6073d41327c486a240046fc1a0a9
Jun 20 19:05:10 dist-test-slave-6wrf krb5kdc[4041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982310, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:10.658671 4125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:10.643295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:50223 (local address 127.1.154.190:46559)
0620 19:05:10.643450 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:10.643454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:10.644473 (+ 1019us) server_negotiation.cc:407] Connection header received
0620 19:05:10.646883 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:10.646888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:10.646988 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:10.647288 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:10.649056 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:10.649820 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:10.651180 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:10.651401 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:10.651495 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:10.654135 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:10.654139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:10.654141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:10.654166 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:10.655785 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:10.656333 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:10.656336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:10.656338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:10.656434 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:10.656889 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:10.656891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:10.656893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:10.657083 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:10.657171 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:10.658295 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0620 19:05:10.658469 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 19:05:10.659968 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:10.643572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:46559 (local address 127.1.154.131:50223)
0620 19:05:10.644213 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:10.644286 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:10.646540 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:10.647448 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:10.647460 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:10.648059 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:10.648825 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:10.648848 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:10.650026 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:10.650033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:10.651043 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:10.651054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:10.651546 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:10.651549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:10.651689 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:10.652323 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:10.652341 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:10.653953 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:10.655952 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:10.655956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:10.655969 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:10.656220 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:10.656539 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:10.656541 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:10.656543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:10.656674 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:10.657307 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:10.657317 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:10.657881 (+ 564us) client_negotiation.cc:769] Sending connection context
0620 19:05:10.658169 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 19:05:10.658558 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":106,"threads_started":1}
I20260620 19:05:10.662497 4545 heartbeater.cc:347] Connected to a master server at 127.1.154.190:46559
I20260620 19:05:10.662792 4545 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:10.663381 4545 heartbeater.cc:511] Master 127.1.154.190:46559 requested a full tablet report, sending...
I20260620 19:05:10.664659 4082 ts_manager.cc:194] Registered new tserver with Master: c567c7167acd464285db0da79512b9cd (127.1.154.131:35905)
I20260620 19:05:10.665468 4082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:50223
I20260620 19:05:10.677402 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:05:10.692701 4125 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:10.690436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50486 (local address 127.1.154.190:46559)
0620 19:05:10.690816 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:10.690822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:10.691014 (+ 192us) server_negotiation.cc:407] Connection header received
0620 19:05:10.691267 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:10.691270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:10.691324 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:10.691423 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:10.692536 (+ 1113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50486: BlockingRecv error: recv got EOF from 127.0.0.1:50486 (error 108)
Metrics: {"server-negotiator.queue_time_us":221}
I20260620 19:05:10.694010 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 4148
I20260620 19:05:10.699680 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 4282
I20260620 19:05:10.705144 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 4416
I20260620 19:05:10.711483 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 4054
2026-06-20T19:05:10Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2353 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:05:10 dist-test-slave-6wrf krb5kdc[4557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:05:10 dist-test-slave-6wrf krb5kdc[4557](info): set up 2 sockets
Jun 20 19:05:10 dist-test-slave-6wrf krb5kdc[4557](info): commencing operation
krb5kdc: starting...
W20260620 19:05:11.797675 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:05:11 dist-test-slave-6wrf krb5kdc[4557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:05:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:05:11.934937 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43323
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:46115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:43323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:12.051956 4570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:12.052538 4570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:12.052665 4570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:12.057706 4570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:12.057821 4570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:12.057862 4570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:12.057901 4570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:12.057931 4570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:12.063871 4570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:46115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:43323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:12.065034 4570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:12.066121 4570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:05:12.075532 4570 server_base.cc:1034] running on GCE node
W20260620 19:05:12.075490 4578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:12.075524 4575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:12.075490 4576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:12.076299 4570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:12.077458 4570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:12.078656 4570 hybrid_clock.cc:648] HybridClock initialized: now 1781982312078645 us; error 30 us; skew 500 ppm
Jun 20 19:05:12 dist-test-slave-6wrf krb5kdc[4557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982312, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:12.089407 4570 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:05:12.091132 4570 webserver.cc:460] Webserver started at http://127.1.154.190:41699/ using document root <none> and password file <none>
I20260620 19:05:12.091943 4570 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:12.092041 4570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:12.092295 4570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:12.095117 4570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "1fa3244e47064b12a6110240f89de455"
format_stamp: "Formatted at 2026-06-20 19:05:12 on dist-test-slave-6wrf"
server_key: "39a0ae61a922ca823dd2572113b0629d"
server_key_iv: "a0508435cd100a5f97aa56c179b2e6dc"
server_key_version: "encryptionkey@0"
I20260620 19:05:12.095800 4570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "1fa3244e47064b12a6110240f89de455"
format_stamp: "Formatted at 2026-06-20 19:05:12 on dist-test-slave-6wrf"
server_key: "39a0ae61a922ca823dd2572113b0629d"
server_key_iv: "a0508435cd100a5f97aa56c179b2e6dc"
server_key_version: "encryptionkey@0"
I20260620 19:05:12.099977 4570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:05:12.102573 4585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:12.104410 4570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:05:12.104696 4570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "1fa3244e47064b12a6110240f89de455"
format_stamp: "Formatted at 2026-06-20 19:05:12 on dist-test-slave-6wrf"
server_key: "39a0ae61a922ca823dd2572113b0629d"
server_key_iv: "a0508435cd100a5f97aa56c179b2e6dc"
server_key_version: "encryptionkey@0"
I20260620 19:05:12.104866 4570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:12.118099 4570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:12.122947 4570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:12.123235 4570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:12.132735 4570 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:43323
I20260620 19:05:12.133008 4637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:43323 every 8 connection(s)
I20260620 19:05:12.134064 4570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:12.136950 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:12.142737 4638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455: Bootstrap starting.
I20260620 19:05:12.143821 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 4570
I20260620 19:05:12.143924 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:12.144198 1642 external_mini_cluster.cc:1428] Setting key 138a844b8308e0a817f87d0b399a48b7
I20260620 19:05:12.145766 4638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:12.146509 4638 log.cc:824] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:12.148283 4638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455: No bootstrap required, opened a new log
I20260620 19:05:12.151746 4638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa3244e47064b12a6110240f89de455" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43323 } }
I20260620 19:05:12.151999 4638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:12.152079 4638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa3244e47064b12a6110240f89de455, State: Initialized, Role: FOLLOWER
I20260620 19:05:12.152549 4638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa3244e47064b12a6110240f89de455" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43323 } }
I20260620 19:05:12.152694 4638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:12.152787 4638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:12.152887 4638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 19:05:12 dist-test-slave-6wrf krb5kdc[4557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:12.154003 4638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa3244e47064b12a6110240f89de455" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43323 } }
I20260620 19:05:12.154343 4638 leader_election.cc:302] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fa3244e47064b12a6110240f89de455; no voters:
I20260620 19:05:12.154698 4638 leader_election.cc:288] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:12.154906 4643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:12.155121 4643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [term 1 LEADER]: Becoming Leader. State: Replica: 1fa3244e47064b12a6110240f89de455, State: Running, Role: LEADER
I20260620 19:05:12.155429 4643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa3244e47064b12a6110240f89de455" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43323 } }
I20260620 19:05:12.156060 4638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:12.158097 4644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fa3244e47064b12a6110240f89de455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa3244e47064b12a6110240f89de455" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43323 } } }
I20260620 19:05:12.158303 4644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:12.158416 4645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fa3244e47064b12a6110240f89de455. Latest consensus state: current_term: 1 leader_uuid: "1fa3244e47064b12a6110240f89de455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa3244e47064b12a6110240f89de455" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43323 } } }
I20260620 19:05:12.158529 4645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:12.160384 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:12.145233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.1.154.190:43323)
0620 19:05:12.145668 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:12.145683 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:12.145717 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:05:12.146379 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:12.146395 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:12.146758 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:12.147079 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:12.147891 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:12.148901 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:12.149804 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:12.150449 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:12.150803 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:12.154196 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:12.154202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:12.154215 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:12.154247 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:12.156377 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:12.156952 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:12.156962 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:12.156972 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:12.157130 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:12.157469 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:12.157473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:12.157474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:12.158029 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:12.158234 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:12.159152 (+ 918us) server_negotiation.cc:299] Negotiation successful
0620 19:05:12.159455 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":135,"threads_started":1}
W20260620 19:05:12.161170 4659 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:05:12.162195 4659 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:05:12.162413 4658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:12.163926 4658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:12.164705 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:05:12.169555 4658 catalog_manager.cc:1347] Generated new cluster ID: 523181253a7e443083c42cc5bf62d421
I20260620 19:05:12.169636 4658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:12.184331 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:12.178750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.1.154.190:43323)
0620 19:05:12.178859 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:12.178864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:12.179120 (+ 256us) server_negotiation.cc:407] Connection header received
0620 19:05:12.179243 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:12.179246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:12.179301 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:12.179394 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:12.179945 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:12.180732 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:12.181892 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:12.182264 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:12.182372 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:12.183279 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:12.183282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:12.183285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:12.183883 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:12.183915 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:12.183934 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:12.184022 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:12.184123 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 19:05:12.184180 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:05:12.201630 4658 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:12.202631 4658 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:12.207942 4658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fa3244e47064b12a6110240f89de455: Generated new TSK 0
I20260620 19:05:12.208626 4658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:05:12.217173 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 4570
2026-06-20T19:05:12Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1500 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:05:12 dist-test-slave-6wrf krb5kdc[4671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:05:12 dist-test-slave-6wrf krb5kdc[4671](info): set up 2 sockets
Jun 20 19:05:12 dist-test-slave-6wrf krb5kdc[4671](info): commencing operation
krb5kdc: starting...
W20260620 19:05:13.290637 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:05:13 dist-test-slave-6wrf krb5kdc[4671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:05:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:05:13.426097 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:40009
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:40009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:13.537005 4684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:13.537330 4684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:13.537405 4684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:13.541443 4684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:13.541539 4684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:13.541579 4684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:13.541666 4684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:13.541697 4684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:13.546983 4684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:40009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:40009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:13.548064 4684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:13.549072 4684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:13.558799 4692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:13.558816 4689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:13.558856 4684 server_base.cc:1034] running on GCE node
W20260620 19:05:13.558820 4690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:13.559535 4684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:13.560672 4684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:13.561869 4684 hybrid_clock.cc:648] HybridClock initialized: now 1781982313561829 us; error 59 us; skew 500 ppm
Jun 20 19:05:13 dist-test-slave-6wrf krb5kdc[4671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982313, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:13.565052 4684 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:05:13.566556 4684 webserver.cc:460] Webserver started at http://127.1.154.190:36943/ using document root <none> and password file <none>
I20260620 19:05:13.567400 4684 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:13.567512 4684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:13.567801 4684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:13.570787 4684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "4c3e36bba2a847738383e9e32dda5ec3"
format_stamp: "Formatted at 2026-06-20 19:05:13 on dist-test-slave-6wrf"
server_key: "2b74a3847b9fc7254ad1ace78210eb17"
server_key_iv: "8b6862a2c5ab519e866d2d1faec847e1"
server_key_version: "encryptionkey@0"
I20260620 19:05:13.571550 4684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "4c3e36bba2a847738383e9e32dda5ec3"
format_stamp: "Formatted at 2026-06-20 19:05:13 on dist-test-slave-6wrf"
server_key: "2b74a3847b9fc7254ad1ace78210eb17"
server_key_iv: "8b6862a2c5ab519e866d2d1faec847e1"
server_key_version: "encryptionkey@0"
I20260620 19:05:13.575997 4684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:05:13.578765 4699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:13.580420 4684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 19:05:13.580565 4684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "4c3e36bba2a847738383e9e32dda5ec3"
format_stamp: "Formatted at 2026-06-20 19:05:13 on dist-test-slave-6wrf"
server_key: "2b74a3847b9fc7254ad1ace78210eb17"
server_key_iv: "8b6862a2c5ab519e866d2d1faec847e1"
server_key_version: "encryptionkey@0"
I20260620 19:05:13.580708 4684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:13.597922 4684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:13.602593 4684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:13.602937 4684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:13.610320 4684 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:40009
I20260620 19:05:13.610652 4751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:40009 every 8 connection(s)
I20260620 19:05:13.611327 4684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:13.614341 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 4684
I20260620 19:05:13.614497 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:13.614832 1642 external_mini_cluster.cc:1428] Setting key 015e89ae51b5ed0f60fb86cda83ac13d
I20260620 19:05:13.615447 4752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:13.622550 4752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3: Bootstrap starting.
I20260620 19:05:13.625234 4752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3: Neither blocks nor log segments found. Creating new log.
Jun 20 19:05:13 dist-test-slave-6wrf krb5kdc[4671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:13.626061 4752 log.cc:824] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:13.628018 4752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3: No bootstrap required, opened a new log
I20260620 19:05:13.631134 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:13.616403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.1.154.190:40009)
0620 19:05:13.617071 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:13.617105 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:13.617156 (+ 51us) server_negotiation.cc:407] Connection header received
0620 19:05:13.618291 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:13.618316 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:13.618795 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:13.619297 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:13.620222 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:13.621200 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:13.622400 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:13.623224 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:13.623410 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:13.626346 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:13.626354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:13.626370 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:13.626407 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:13.628105 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:13.628642 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:13.628647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:13.628651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:13.628774 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:13.629115 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:13.629118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:13.629120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:13.629527 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:13.629800 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:13.630093 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 19:05:13.630394 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":243,"threads_started":1}
I20260620 19:05:13.632081 4752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e36bba2a847738383e9e32dda5ec3" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40009 } }
I20260620 19:05:13.632364 4752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:13.632414 4752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c3e36bba2a847738383e9e32dda5ec3, State: Initialized, Role: FOLLOWER
I20260620 19:05:13.632876 4752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e36bba2a847738383e9e32dda5ec3" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40009 } }
I20260620 19:05:13.633033 4752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:13.633087 4752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:13.633203 4752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:13.634259 4752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e36bba2a847738383e9e32dda5ec3" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40009 } }
I20260620 19:05:13.634559 4752 leader_election.cc:302] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c3e36bba2a847738383e9e32dda5ec3; no voters:
I20260620 19:05:13.634827 4752 leader_election.cc:288] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:13.634980 4757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:13.635228 4757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [term 1 LEADER]: Becoming Leader. State: Replica: 4c3e36bba2a847738383e9e32dda5ec3, State: Running, Role: LEADER
I20260620 19:05:13.635529 4757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e36bba2a847738383e9e32dda5ec3" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40009 } }
I20260620 19:05:13.636098 4752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:13.637140 4758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c3e36bba2a847738383e9e32dda5ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e36bba2a847738383e9e32dda5ec3" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40009 } } }
I20260620 19:05:13.637271 4758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:13.637691 4759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c3e36bba2a847738383e9e32dda5ec3. Latest consensus state: current_term: 1 leader_uuid: "4c3e36bba2a847738383e9e32dda5ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e36bba2a847738383e9e32dda5ec3" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40009 } } }
I20260620 19:05:13.638355 4759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:13.638351 4766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:13.641330 4766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:13.647360 4766 catalog_manager.cc:1347] Generated new cluster ID: 6b3938bfd2b74d239b08f0112bc75b55
I20260620 19:05:13.647445 4766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:13.668439 4766 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:13.669790 4766 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:13.680409 4766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c3e36bba2a847738383e9e32dda5ec3: Generated new TSK 0
I20260620 19:05:13.681608 4766 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:05:13.712937 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40009
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:13.855789 4778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:13.856076 4778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:13.856153 4778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:13.860266 4778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:13.860328 4778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:13.860388 4778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:05:13.865762 4778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:13.866807 4778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:13.867789 4778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:05:13.877871 4778 server_base.cc:1034] running on GCE node
W20260620 19:05:13.877848 4784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:13.877889 4783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:13.877838 4786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:13.878502 4778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:13.879127 4778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:13.882426 4778 hybrid_clock.cc:648] HybridClock initialized: now 1781982313882411 us; error 58 us; skew 500 ppm
Jun 20 19:05:13 dist-test-slave-6wrf krb5kdc[4671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982313, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:13.885708 4778 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:05:13.886677 4778 webserver.cc:460] Webserver started at http://127.1.154.129:38563/ using document root <none> and password file <none>
I20260620 19:05:13.887256 4778 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:13.887344 4778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:13.887573 4778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:13.889791 4778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "2cc47ef112e0447e9b2205a43fa3ef2d"
format_stamp: "Formatted at 2026-06-20 19:05:13 on dist-test-slave-6wrf"
server_key: "ee25fc3bc8edbc0a0040c1fd3c61508c"
server_key_iv: "4b387e0d90db5bd2239b1fa9db86f572"
server_key_version: "encryptionkey@0"
I20260620 19:05:13.890376 4778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "2cc47ef112e0447e9b2205a43fa3ef2d"
format_stamp: "Formatted at 2026-06-20 19:05:13 on dist-test-slave-6wrf"
server_key: "ee25fc3bc8edbc0a0040c1fd3c61508c"
server_key_iv: "4b387e0d90db5bd2239b1fa9db86f572"
server_key_version: "encryptionkey@0"
I20260620 19:05:13.894194 4778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:05:13.896512 4793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:13.897769 4778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:05:13.897897 4778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "2cc47ef112e0447e9b2205a43fa3ef2d"
format_stamp: "Formatted at 2026-06-20 19:05:13 on dist-test-slave-6wrf"
server_key: "ee25fc3bc8edbc0a0040c1fd3c61508c"
server_key_iv: "4b387e0d90db5bd2239b1fa9db86f572"
server_key_version: "encryptionkey@0"
I20260620 19:05:13.898005 4778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:13.915326 4778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:13.918690 4778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:13.918895 4778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:13.919477 4778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:13.920334 4778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:13.920401 4778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:13.920466 4778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:13.920513 4778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:13.930145 4778 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:32945
I20260620 19:05:13.930186 4906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:32945 every 8 connection(s)
I20260620 19:05:13.931032 4778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:05:13 dist-test-slave-6wrf krb5kdc[4671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982313, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:13.941315 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 4778
I20260620 19:05:13.941588 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:05:13.942008 1642 external_mini_cluster.cc:1428] Setting key c40fd611e2c796202a6aebd7164b7aa6
I20260620 19:05:13.946547 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:13.932873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:33549 (local address 127.1.154.190:40009)
0620 19:05:13.933014 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:13.933020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:13.933784 (+ 764us) server_negotiation.cc:407] Connection header received
0620 19:05:13.935171 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:13.935175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:13.935227 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:13.935305 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:13.936539 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:13.937151 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:13.938452 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:13.938686 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:13.938796 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:13.941907 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:13.941910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:13.941912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:13.941934 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:13.943690 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:13.944249 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:13.944252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:13.944254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:13.944335 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:13.945134 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:13.945138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:13.945141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:13.945506 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:13.945702 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:13.946243 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 19:05:13.946374 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:05:13.947873 4909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:13.933140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:40009 (local address 127.1.154.129:33549)
0620 19:05:13.933581 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:13.933690 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:13.934963 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:13.935460 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:13.935466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:13.935863 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:13.936358 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:13.936378 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:13.937280 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:13.937284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:13.938333 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:13.938342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:13.938832 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:13.938836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:13.939002 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:13.939606 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:13.939623 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:13.941737 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:13.943835 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:13.943844 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:13.943861 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:13.944120 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:13.944430 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:13.944435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:13.944437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:13.944937 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:13.945697 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:13.945704 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:13.945968 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 19:05:13.946186 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 19:05:13.946509 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":103,"threads_started":1}
I20260620 19:05:13.949867 4907 heartbeater.cc:347] Connected to a master server at 127.1.154.190:40009
I20260620 19:05:13.950258 4907 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:13.950883 4907 heartbeater.cc:511] Master 127.1.154.190:40009 requested a full tablet report, sending...
I20260620 19:05:13.952621 4716 ts_manager.cc:194] Registered new tserver with Master: 2cc47ef112e0447e9b2205a43fa3ef2d (127.1.154.129:32945)
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
I20260620 19:05:13.953850 4716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:33549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:05:13.975242 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40009
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:14.087374 4912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:14.087651 4912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:14.087695 4912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:14.091853 4912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:14.091914 4912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:14.091972 4912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:14.097327 4912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:14.098405 4912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:14.099378 4912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:14.109445 4917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:14.109496 4918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:14.109445 4920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:14.109515 4912 server_base.cc:1034] running on GCE node
I20260620 19:05:14.110574 4912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:14.111259 4912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:14.112476 4912 hybrid_clock.cc:648] HybridClock initialized: now 1781982314112450 us; error 63 us; skew 500 ppm
Jun 20 19:05:14 dist-test-slave-6wrf krb5kdc[4671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982314, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:14.115360 4912 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:05:14.116261 4912 webserver.cc:460] Webserver started at http://127.1.154.130:45307/ using document root <none> and password file <none>
I20260620 19:05:14.116771 4912 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:14.116827 4912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:14.116998 4912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:14.119038 4912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "524c0a8e097c4e2bbb8924c1f159fa4f"
format_stamp: "Formatted at 2026-06-20 19:05:14 on dist-test-slave-6wrf"
server_key: "48bc8566fba6e91fa5326dddddca2040"
server_key_iv: "1475d333da131272f69116de555fc161"
server_key_version: "encryptionkey@0"
I20260620 19:05:14.119552 4912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "524c0a8e097c4e2bbb8924c1f159fa4f"
format_stamp: "Formatted at 2026-06-20 19:05:14 on dist-test-slave-6wrf"
server_key: "48bc8566fba6e91fa5326dddddca2040"
server_key_iv: "1475d333da131272f69116de555fc161"
server_key_version: "encryptionkey@0"
I20260620 19:05:14.123163 4912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:05:14.125885 4927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:14.127226 4912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:05:14.127352 4912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "524c0a8e097c4e2bbb8924c1f159fa4f"
format_stamp: "Formatted at 2026-06-20 19:05:14 on dist-test-slave-6wrf"
server_key: "48bc8566fba6e91fa5326dddddca2040"
server_key_iv: "1475d333da131272f69116de555fc161"
server_key_version: "encryptionkey@0"
I20260620 19:05:14.127461 4912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:14.145821 4912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:14.149036 4912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:14.149194 4912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:14.149886 4912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:14.151212 4912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:14.151286 4912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:14.151487 4912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:14.151572 4912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:14.160354 4912 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:35921
I20260620 19:05:14.160379 5040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:35921 every 8 connection(s)
I20260620 19:05:14.161217 4912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:14.162051 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 4912
I20260620 19:05:14.162178 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:05:14.162472 1642 external_mini_cluster.cc:1428] Setting key 6296af4cd18cc3358f1847f7f7e00a6a
Jun 20 19:05:14 dist-test-slave-6wrf krb5kdc[4671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982314, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
I20260620 19:05:14.177659 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:14.163229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:46525 (local address 127.1.154.190:40009)
0620 19:05:14.163363 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:14.163367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:14.164306 (+ 939us) server_negotiation.cc:407] Connection header received
0620 19:05:14.166309 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:14.166312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:14.166364 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:14.166442 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:14.167830 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:14.168634 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:14.170474 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:14.170926 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:14.171071 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:14.174078 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:14.174082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:14.174085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:14.174108 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:14.175487 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:14.176043 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:14.176047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:14.176048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:14.176131 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:14.176484 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:14.176487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:14.176488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:14.176682 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:14.176789 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:14.177330 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 19:05:14.177451 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:05:14.178486 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:14.163602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:40009 (local address 127.1.154.130:46525)
0620 19:05:14.164140 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:14.164195 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:14.166102 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:14.166623 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:14.166629 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:14.166978 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:14.167556 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:14.167578 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:14.168837 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:14.168840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:14.170294 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:14.170305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:14.171155 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:14.171163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:14.171448 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:14.172199 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:14.172220 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:14.173898 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:14.175628 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:14.175634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:14.175649 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:14.175919 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:14.176238 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:14.176241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:14.176243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:14.176382 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:14.176787 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:14.176794 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:14.177072 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 19:05:14.177276 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 19:05:14.177636 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":140,"threads_started":1}
I20260620 19:05:14.180007 5041 heartbeater.cc:347] Connected to a master server at 127.1.154.190:40009
I20260620 19:05:14.180296 5041 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:14.180876 5041 heartbeater.cc:511] Master 127.1.154.190:40009 requested a full tablet report, sending...
I20260620 19:05:14.182085 4716 ts_manager.cc:194] Registered new tserver with Master: 524c0a8e097c4e2bbb8924c1f159fa4f (127.1.154.130:35921)
I20260620 19:05:14.183025 4716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:46525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:05:14.194536 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40009
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:14.305637 5046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:14.305897 5046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:14.305945 5046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:14.309944 5046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:14.310010 5046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:14.310070 5046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:14.315338 5046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:14.316322 5046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:14.317351 5046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:05:14.327309 5046 server_base.cc:1034] running on GCE node
W20260620 19:05:14.327564 5051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:14.327335 5054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:14.327334 5052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:14.328274 5046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:14.328945 5046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:14.330168 5046 hybrid_clock.cc:648] HybridClock initialized: now 1781982314330154 us; error 26 us; skew 500 ppm
Jun 20 19:05:14 dist-test-slave-6wrf krb5kdc[4671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982314, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:14.333062 5046 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:05:14.334010 5046 webserver.cc:460] Webserver started at http://127.1.154.131:44417/ using document root <none> and password file <none>
I20260620 19:05:14.334573 5046 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:14.334635 5046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:14.334846 5046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:14.337438 5046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "b1cf9e78a1a948fbb19834aad4a341bd"
format_stamp: "Formatted at 2026-06-20 19:05:14 on dist-test-slave-6wrf"
server_key: "f650564a58bc2b8fe8f17616e2adb5d0"
server_key_iv: "45e3a4168b18caca9739763cafa10990"
server_key_version: "encryptionkey@0"
I20260620 19:05:14.338168 5046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "b1cf9e78a1a948fbb19834aad4a341bd"
format_stamp: "Formatted at 2026-06-20 19:05:14 on dist-test-slave-6wrf"
server_key: "f650564a58bc2b8fe8f17616e2adb5d0"
server_key_iv: "45e3a4168b18caca9739763cafa10990"
server_key_version: "encryptionkey@0"
I20260620 19:05:14.341959 5046 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:05:14.344376 5061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:14.345788 5046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:14.345911 5046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "b1cf9e78a1a948fbb19834aad4a341bd"
format_stamp: "Formatted at 2026-06-20 19:05:14 on dist-test-slave-6wrf"
server_key: "f650564a58bc2b8fe8f17616e2adb5d0"
server_key_iv: "45e3a4168b18caca9739763cafa10990"
server_key_version: "encryptionkey@0"
I20260620 19:05:14.346019 5046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:14.364082 5046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:14.367548 5046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:14.367767 5046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:14.368319 5046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:14.369097 5046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:14.369143 5046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:14.369184 5046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:14.369206 5046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:14.378508 5046 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:45239
I20260620 19:05:14.378583 5174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:45239 every 8 connection(s)
I20260620 19:05:14.379383 5046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:14.381654 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 5046
I20260620 19:05:14.381776 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:05:14.382071 1642 external_mini_cluster.cc:1428] Setting key dc7a7c60729601a5c2db5c3cc8879ffa
Jun 20 19:05:14 dist-test-slave-6wrf krb5kdc[4671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982314, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:14.394383 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:14.381716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:53685 (local address 127.1.154.190:40009)
0620 19:05:14.382137 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:14.382143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:14.383435 (+ 1292us) server_negotiation.cc:407] Connection header received
0620 19:05:14.385051 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:14.385055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:14.385107 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:14.385187 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:14.386449 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:14.386979 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:14.388088 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:14.388300 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:14.388408 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:14.390996 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:14.391000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:14.391002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:14.391026 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:14.392455 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:14.392924 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:14.392926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:14.392927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:14.392999 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:14.393369 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:14.393371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:14.393372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:14.393536 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:14.393678 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:14.394169 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 19:05:14.394262 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247}
I20260620 19:05:14.395388 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:14.382705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:40009 (local address 127.1.154.131:53685)
0620 19:05:14.383288 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:14.383346 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:14.384840 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:14.385379 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:14.385386 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:14.385824 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:14.386306 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:14.386319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:14.387103 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:14.387109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:14.387958 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:14.387967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:14.388446 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:14.388450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:14.388617 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:14.389226 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:14.389243 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:14.390793 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:14.392564 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:14.392568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:14.392580 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:14.392825 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:14.393105 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:14.393108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:14.393109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:14.393253 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:14.393694 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:14.393699 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:14.393938 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 19:05:14.394103 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 19:05:14.394365 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":134,"threads_started":1}
I20260620 19:05:14.396711 5175 heartbeater.cc:347] Connected to a master server at 127.1.154.190:40009
I20260620 19:05:14.397115 5175 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:14.397665 5175 heartbeater.cc:511] Master 127.1.154.190:40009 requested a full tablet report, sending...
I20260620 19:05:14.398676 4716 ts_manager.cc:194] Registered new tserver with Master: b1cf9e78a1a948fbb19834aad4a341bd (127.1.154.131:45239)
I20260620 19:05:14.399256 4716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:53685
I20260620 19:05:14.408634 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:05:14 dist-test-slave-6wrf krb5kdc[4671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982314, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 19:05:14 dist-test-slave-6wrf krb5kdc[4671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982314, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:14.437877 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:14.428671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.1.154.190:40009)
0620 19:05:14.428847 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:14.428852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:14.429019 (+ 167us) server_negotiation.cc:407] Connection header received
0620 19:05:14.429190 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:14.429194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:14.429238 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:14.429324 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:14.430097 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:14.430724 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:14.431588 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:14.431874 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:14.431981 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:14.434569 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:14.434573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:14.434576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:14.434608 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:14.436054 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:14.436497 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:14.436503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:14.436505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:14.436610 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:14.436898 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:14.436903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:14.436904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:14.437201 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:14.437332 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:14.437575 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 19:05:14.437719 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 19:05:14.438516 4716 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47876
I20260620 19:05:14.439622 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 4778
I20260620 19:05:14.445802 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 4912
I20260620 19:05:14.451521 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 5046
I20260620 19:05:14.458133 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 4684
2026-06-20T19:05:14Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2246 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:05:14 dist-test-slave-6wrf krb5kdc[5187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:05:14 dist-test-slave-6wrf krb5kdc[5187](info): set up 2 sockets
Jun 20 19:05:14 dist-test-slave-6wrf krb5kdc[5187](info): commencing operation
krb5kdc: starting...
W20260620 19:05:15.534029 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:05:15 dist-test-slave-6wrf krb5kdc[5187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:05:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:05:15.666854 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:37893
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:37893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:15.789176 5200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:15.789490 5200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:15.789572 5200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:15.793694 5200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:15.793780 5200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:15.793821 5200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:15.793860 5200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:15.793891 5200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:15.799613 5200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:37893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:37893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:15.801247 5200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:15.802707 5200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:15.813560 5205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:15.813621 5206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:15.814028 5200 server_base.cc:1034] running on GCE node
W20260620 19:05:15.813560 5208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:15.814858 5200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:15.816166 5200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:15.817381 5200 hybrid_clock.cc:648] HybridClock initialized: now 1781982315817296 us; error 102 us; skew 500 ppm
Jun 20 19:05:15 dist-test-slave-6wrf krb5kdc[5187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982315, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:15.820966 5200 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:05:15.822376 5200 webserver.cc:460] Webserver started at http://127.1.154.190:34641/ using document root <none> and password file <none>
I20260620 19:05:15.823273 5200 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:15.823432 5200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:15.823719 5200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:15.826006 5200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "aecf86a13e0e4023a6cfa372c7f6e5d9"
format_stamp: "Formatted at 2026-06-20 19:05:15 on dist-test-slave-6wrf"
server_key: "0d4f03f5d382ffc7ec5a5021ab831c11"
server_key_iv: "97ce031abf2f45bacd592d7a47b03bf5"
server_key_version: "encryptionkey@0"
I20260620 19:05:15.826611 5200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "aecf86a13e0e4023a6cfa372c7f6e5d9"
format_stamp: "Formatted at 2026-06-20 19:05:15 on dist-test-slave-6wrf"
server_key: "0d4f03f5d382ffc7ec5a5021ab831c11"
server_key_iv: "97ce031abf2f45bacd592d7a47b03bf5"
server_key_version: "encryptionkey@0"
I20260620 19:05:15.830426 5200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:05:15.832875 5215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:15.834196 5200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:05:15.834352 5200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "aecf86a13e0e4023a6cfa372c7f6e5d9"
format_stamp: "Formatted at 2026-06-20 19:05:15 on dist-test-slave-6wrf"
server_key: "0d4f03f5d382ffc7ec5a5021ab831c11"
server_key_iv: "97ce031abf2f45bacd592d7a47b03bf5"
server_key_version: "encryptionkey@0"
I20260620 19:05:15.834475 5200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:15.865257 5200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:15.868402 5200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:15.868646 5200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:15.876550 5200 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:37893
I20260620 19:05:15.876566 5267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:37893 every 8 connection(s)
I20260620 19:05:15.877538 5200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:15.881395 5268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:15.885727 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 5200
I20260620 19:05:15.885849 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:15.886178 1642 external_mini_cluster.cc:1428] Setting key 276529dff9a8d5edc6707a0b81a9363b
I20260620 19:05:15.888866 5268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9: Bootstrap starting.
I20260620 19:05:15.891146 5268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:15.891865 5268 log.cc:824] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:15.893748 5268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9: No bootstrap required, opened a new log
Jun 20 19:05:15 dist-test-slave-6wrf krb5kdc[5187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:15.897830 5268 raft_consensus.cc:348] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf86a13e0e4023a6cfa372c7f6e5d9" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37893 } }
I20260620 19:05:15.898092 5268 raft_consensus.cc:374] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:15.898180 5268 raft_consensus.cc:729] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aecf86a13e0e4023a6cfa372c7f6e5d9, State: Initialized, Role: FOLLOWER
I20260620 19:05:15.898658 5268 consensus_queue.cc:260] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf86a13e0e4023a6cfa372c7f6e5d9" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37893 } }
I20260620 19:05:15.898808 5268 raft_consensus.cc:388] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:15.898876 5268 raft_consensus.cc:482] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:15.898998 5268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:15.900054 5268 raft_consensus.cc:504] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf86a13e0e4023a6cfa372c7f6e5d9" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37893 } }
I20260620 19:05:15.900401 5268 leader_election.cc:302] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aecf86a13e0e4023a6cfa372c7f6e5d9; no voters:
I20260620 19:05:15.900758 5268 leader_election.cc:288] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:15.900825 5273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:15.901012 5273 raft_consensus.cc:686] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [term 1 LEADER]: Becoming Leader. State: Replica: aecf86a13e0e4023a6cfa372c7f6e5d9, State: Running, Role: LEADER
I20260620 19:05:15.901455 5273 consensus_queue.cc:237] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf86a13e0e4023a6cfa372c7f6e5d9" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37893 } }
I20260620 19:05:15.902160 5268 sys_catalog.cc:564] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:15.903447 5275 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aecf86a13e0e4023a6cfa372c7f6e5d9. Latest consensus state: current_term: 1 leader_uuid: "aecf86a13e0e4023a6cfa372c7f6e5d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf86a13e0e4023a6cfa372c7f6e5d9" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37893 } } }
I20260620 19:05:15.903574 5275 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:15.903913 5274 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aecf86a13e0e4023a6cfa372c7f6e5d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf86a13e0e4023a6cfa372c7f6e5d9" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37893 } } }
I20260620 19:05:15.903995 5274 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:15.903998 5282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:15.904526 5271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:15.887782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45768 (local address 127.1.154.190:37893)
0620 19:05:15.888259 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:15.888274 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:15.888313 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:05:15.889033 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:15.889067 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:15.889408 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:15.889812 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:15.890622 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:15.891702 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:15.892597 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:15.893066 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:15.893247 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:15.896142 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:15.896147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:15.896161 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:15.896187 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:15.898735 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:15.899388 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:15.899396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:15.899407 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:15.899898 (+ 491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:15.900457 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:15.900462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:15.900465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:15.901417 (+ 952us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:15.902515 (+ 1098us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:15.902858 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 19:05:15.903433 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":146,"threads_started":1}
I20260620 19:05:15.908558 5282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:15.914868 5282 catalog_manager.cc:1347] Generated new cluster ID: d756b1136a4b4419878bf54a3bba5f15
I20260620 19:05:15.914959 5282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:15.921732 5282 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:15.923511 5282 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:15.929426 5282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aecf86a13e0e4023a6cfa372c7f6e5d9: Generated new TSK 0
I20260620 19:05:15.930639 5282 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:05:15.969483 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:37893
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:16.095777 5294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:16.096197 5294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:16.096294 5294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:16.101130 5294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:16.101195 5294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:16.101264 5294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:05:16.107182 5294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:37893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:16.108201 5294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:16.109475 5294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:16.120769 5302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:16.120915 5300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:16.120795 5299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:16.121308 5294 server_base.cc:1034] running on GCE node
I20260620 19:05:16.121986 5294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:16.123143 5294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:16.124392 5294 hybrid_clock.cc:648] HybridClock initialized: now 1781982316124370 us; error 38 us; skew 500 ppm
Jun 20 19:05:16 dist-test-slave-6wrf krb5kdc[5187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:16.127737 5294 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:05:16.128690 5294 webserver.cc:460] Webserver started at http://127.1.154.129:33525/ using document root <none> and password file <none>
I20260620 19:05:16.129271 5294 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:16.129361 5294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:16.129587 5294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:16.131729 5294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "89d6d050a27b47daa539b3daaa46692f"
format_stamp: "Formatted at 2026-06-20 19:05:16 on dist-test-slave-6wrf"
server_key: "80191ac5ba3bc2c1f99c1b7873707c06"
server_key_iv: "03faec02c2919ff1fee464503c27afdb"
server_key_version: "encryptionkey@0"
I20260620 19:05:16.132426 5294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "89d6d050a27b47daa539b3daaa46692f"
format_stamp: "Formatted at 2026-06-20 19:05:16 on dist-test-slave-6wrf"
server_key: "80191ac5ba3bc2c1f99c1b7873707c06"
server_key_iv: "03faec02c2919ff1fee464503c27afdb"
server_key_version: "encryptionkey@0"
I20260620 19:05:16.136561 5294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 19:05:16.140125 5309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:16.141330 5294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:16.141491 5294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "89d6d050a27b47daa539b3daaa46692f"
format_stamp: "Formatted at 2026-06-20 19:05:16 on dist-test-slave-6wrf"
server_key: "80191ac5ba3bc2c1f99c1b7873707c06"
server_key_iv: "03faec02c2919ff1fee464503c27afdb"
server_key_version: "encryptionkey@0"
I20260620 19:05:16.141665 5294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:16.162624 5294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:16.165936 5294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:16.166167 5294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:16.166757 5294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:16.167598 5294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:16.167671 5294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:16.167745 5294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:16.167789 5294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:16.176575 5294 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:40129
I20260620 19:05:16.176615 5422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:40129 every 8 connection(s)
I20260620 19:05:16.177457 5294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:05:16.187492 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 5294
I20260620 19:05:16.187611 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:05:16.187963 1642 external_mini_cluster.cc:1428] Setting key aa3330ef9011e8ebd3b63152595a562c
Jun 20 19:05:16 dist-test-slave-6wrf krb5kdc[5187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:16.193156 5271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:16.180889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:47151 (local address 127.1.154.190:37893)
0620 19:05:16.181086 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:16.181091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:16.181116 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:05:16.182378 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:16.182383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:16.182474 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:16.182577 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:16.184010 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.184561 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.185780 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.186017 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.186110 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:16.188903 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:16.188908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:16.188912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:16.188985 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:16.190796 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:16.191484 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:16.191490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:16.191494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:16.191610 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:16.191982 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:16.191986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:16.191988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:16.192269 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:16.192380 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:16.192890 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 19:05:16.193028 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 19:05:16.194357 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:16.179946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:37893 (local address 127.1.154.129:47151)
0620 19:05:16.180411 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:16.180462 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:16.182131 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:16.182784 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:16.182793 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:16.183355 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:16.183827 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:16.183840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.184708 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.184712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:16.185657 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:16.185666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.186170 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.186173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:16.186311 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:16.186980 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:16.186997 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:16.188615 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:16.190938 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:16.190947 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:16.190966 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:16.191346 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:16.191744 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:16.191746 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:16.191748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:16.191879 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:16.192374 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:16.192378 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:16.192610 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 19:05:16.192810 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 19:05:16.193187 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":135,"threads_started":1}
I20260620 19:05:16.195675 5423 heartbeater.cc:347] Connected to a master server at 127.1.154.190:37893
I20260620 19:05:16.195931 5423 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:16.196489 5423 heartbeater.cc:511] Master 127.1.154.190:37893 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
I20260620 19:05:16.198462 5229 ts_manager.cc:194] Registered new tserver with Master: 89d6d050a27b47daa539b3daaa46692f (127.1.154.129:40129)
I20260620 19:05:16.200313 5229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:47151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:05:16.217299 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:37893
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:16.340720 5428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:16.341027 5428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:16.341097 5428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:16.345285 5428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:16.345360 5428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:16.345443 5428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:16.350836 5428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:37893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:16.351923 5428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:16.352936 5428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:16.363871 5434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:16.363857 5433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:16.363858 5436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:16.364169 5428 server_base.cc:1034] running on GCE node
I20260620 19:05:16.364601 5428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:16.365268 5428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:16.366452 5428 hybrid_clock.cc:648] HybridClock initialized: now 1781982316366428 us; error 36 us; skew 500 ppm
Jun 20 19:05:16 dist-test-slave-6wrf krb5kdc[5187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:16.369439 5428 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:05:16.370378 5428 webserver.cc:460] Webserver started at http://127.1.154.130:44981/ using document root <none> and password file <none>
I20260620 19:05:16.370970 5428 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:16.371035 5428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:16.371260 5428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:16.373332 5428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "fb5099c014914b0fba9d47088a368c1b"
format_stamp: "Formatted at 2026-06-20 19:05:16 on dist-test-slave-6wrf"
server_key: "66d63747642dcca183c3872288d42561"
server_key_iv: "0e360c27027009b67d0156f5f51daeaf"
server_key_version: "encryptionkey@0"
I20260620 19:05:16.374019 5428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "fb5099c014914b0fba9d47088a368c1b"
format_stamp: "Formatted at 2026-06-20 19:05:16 on dist-test-slave-6wrf"
server_key: "66d63747642dcca183c3872288d42561"
server_key_iv: "0e360c27027009b67d0156f5f51daeaf"
server_key_version: "encryptionkey@0"
I20260620 19:05:16.378479 5428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:05:16.381907 5443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:16.383656 5428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:05:16.383891 5428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "fb5099c014914b0fba9d47088a368c1b"
format_stamp: "Formatted at 2026-06-20 19:05:16 on dist-test-slave-6wrf"
server_key: "66d63747642dcca183c3872288d42561"
server_key_iv: "0e360c27027009b67d0156f5f51daeaf"
server_key_version: "encryptionkey@0"
I20260620 19:05:16.384087 5428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:16.401361 5428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:16.405241 5428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:16.405488 5428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:16.406108 5428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:16.406942 5428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:16.407012 5428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:16.407092 5428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:16.407130 5428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:16.416589 5428 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:46353
I20260620 19:05:16.416612 5556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:46353 every 8 connection(s)
I20260620 19:05:16.417438 5428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:16.425412 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 5428
I20260620 19:05:16.425576 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:05:16.425876 1642 external_mini_cluster.cc:1428] Setting key 4cfc1d6d4e07e68ba9e9ad08a2fe0f4b
Jun 20 19:05:16 dist-test-slave-6wrf krb5kdc[5187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:16.434993 5271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:16.419723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:37817 (local address 127.1.154.190:37893)
0620 19:05:16.419947 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:16.419951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:16.421214 (+ 1263us) server_negotiation.cc:407] Connection header received
0620 19:05:16.422650 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:16.422653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:16.422700 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:16.422785 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:16.424011 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.424539 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.426300 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.426522 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.426648 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:16.429426 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:16.429432 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:16.429437 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:16.429466 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:16.431877 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:16.432886 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:16.432891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:16.432893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:16.432992 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:16.433451 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:16.433456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:16.433461 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:16.433753 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:16.433874 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:16.434655 (+ 781us) server_negotiation.cc:299] Negotiation successful
0620 19:05:16.434791 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
I20260620 19:05:16.436197 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:16.420393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:37893 (local address 127.1.154.130:37817)
0620 19:05:16.421029 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:16.421072 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:16.422431 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:16.422979 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:16.422986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:16.423370 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:16.423870 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:16.423882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.424756 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.424760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:16.426157 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:16.426170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.426688 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.426692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:16.426867 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:16.427471 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:16.427487 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:16.429271 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:16.432090 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:16.432099 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:16.432120 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:16.432669 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:16.433146 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:16.433150 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:16.433152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:16.433330 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:16.433928 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:16.433936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:16.434283 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 19:05:16.434568 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 19:05:16.434952 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":222,"threads_started":1}
I20260620 19:05:16.437863 5557 heartbeater.cc:347] Connected to a master server at 127.1.154.190:37893
I20260620 19:05:16.438216 5557 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
I20260620 19:05:16.439177 5557 heartbeater.cc:511] Master 127.1.154.190:37893 requested a full tablet report, sending...
I20260620 19:05:16.440436 5229 ts_manager.cc:194] Registered new tserver with Master: fb5099c014914b0fba9d47088a368c1b (127.1.154.130:46353)
I20260620 19:05:16.441320 5229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:37817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:05:16.458076 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:37893
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:16.582805 5562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:16.583165 5562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:16.583252 5562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:16.588869 5562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:16.589003 5562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:16.589110 5562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:16.596865 5562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:37893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:16.598935 5562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:16.600456 5562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:16.613530 5568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:16.613821 5567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:16.613821 5570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:16.614051 5562 server_base.cc:1034] running on GCE node
I20260620 19:05:16.614493 5562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:16.615244 5562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:16.616454 5562 hybrid_clock.cc:648] HybridClock initialized: now 1781982316616425 us; error 55 us; skew 500 ppm
Jun 20 19:05:16 dist-test-slave-6wrf krb5kdc[5187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:16.619947 5562 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:05:16.621307 5562 webserver.cc:460] Webserver started at http://127.1.154.131:41225/ using document root <none> and password file <none>
I20260620 19:05:16.622197 5562 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:16.622309 5562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:16.622587 5562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:16.624963 5562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "76c37abd75c24e448bc8e0a7a6505819"
format_stamp: "Formatted at 2026-06-20 19:05:16 on dist-test-slave-6wrf"
server_key: "dd883ef20e14c683efafa864a3047ace"
server_key_iv: "b377c2c11f4bef58f3dc8eb4d20503bf"
server_key_version: "encryptionkey@0"
I20260620 19:05:16.626220 5562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "76c37abd75c24e448bc8e0a7a6505819"
format_stamp: "Formatted at 2026-06-20 19:05:16 on dist-test-slave-6wrf"
server_key: "dd883ef20e14c683efafa864a3047ace"
server_key_iv: "b377c2c11f4bef58f3dc8eb4d20503bf"
server_key_version: "encryptionkey@0"
I20260620 19:05:16.630682 5562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:05:16.633837 5577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:16.635439 5562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 19:05:16.635560 5562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "76c37abd75c24e448bc8e0a7a6505819"
format_stamp: "Formatted at 2026-06-20 19:05:16 on dist-test-slave-6wrf"
server_key: "dd883ef20e14c683efafa864a3047ace"
server_key_iv: "b377c2c11f4bef58f3dc8eb4d20503bf"
server_key_version: "encryptionkey@0"
I20260620 19:05:16.635718 5562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:16.652746 5562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:16.657336 5562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:16.657737 5562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:16.658591 5562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:16.659888 5562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:16.659946 5562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:16.659997 5562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:16.660013 5562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:16.673064 5562 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:43129
I20260620 19:05:16.673089 5690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:43129 every 8 connection(s)
I20260620 19:05:16.674137 5562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:16.681089 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 5562
I20260620 19:05:16.681232 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:05:16.681520 1642 external_mini_cluster.cc:1428] Setting key f7a214d8243eeca9c585824e892e50e4
Jun 20 19:05:16 dist-test-slave-6wrf krb5kdc[5187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:16.689235 5271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:16.676078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:53493 (local address 127.1.154.190:37893)
0620 19:05:16.676247 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:16.676252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:16.677085 (+ 833us) server_negotiation.cc:407] Connection header received
0620 19:05:16.678468 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:16.678470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:16.678526 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:16.678609 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:16.679831 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.680376 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.682003 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.682290 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.682416 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:16.685284 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:16.685287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:16.685289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:16.685310 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:16.686746 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:16.687623 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:16.687625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:16.687627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:16.687700 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:16.688121 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:16.688123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:16.688124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:16.688296 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:16.688389 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:16.689006 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 19:05:16.689112 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 19:05:16.690183 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:16.676458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:37893 (local address 127.1.154.131:53493)
0620 19:05:16.676931 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:16.676973 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:16.678255 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:16.678818 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:16.678825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:16.679196 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:16.679692 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:16.679706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.680505 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.680508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:16.681845 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:16.681858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.682466 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.682469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:16.682725 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:16.683451 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:16.683468 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:16.685125 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:16.686969 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:16.686978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:16.686998 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:16.687439 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:16.687846 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:16.687849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:16.687851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:16.688017 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:16.688415 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:16.688429 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:16.688668 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 19:05:16.688910 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 19:05:16.689216 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":111,"threads_started":1}
I20260620 19:05:16.691275 5691 heartbeater.cc:347] Connected to a master server at 127.1.154.190:37893
I20260620 19:05:16.691488 5691 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:16.691969 5691 heartbeater.cc:511] Master 127.1.154.190:37893 requested a full tablet report, sending...
I20260620 19:05:16.693348 5229 ts_manager.cc:194] Registered new tserver with Master: 76c37abd75c24e448bc8e0a7a6505819 (127.1.154.131:43129)
I20260620 19:05:16.694159 5229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:53493
I20260620 19:05:16.695981 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:05:16 dist-test-slave-6wrf krb5kdc[5187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:05:16 dist-test-slave-6wrf krb5kdc[5187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.129@KRBTEST.COM
I20260620 19:05:16.726768 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:16.715096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.1.154.129:40129)
0620 19:05:16.715399 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:16.715406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:16.715435 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:05:16.715488 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:16.715494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:16.715677 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:16.715799 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:16.716711 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.717883 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.719101 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.719565 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.719688 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:16.722784 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:16.722793 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:16.722807 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:16.722832 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:16.724502 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:16.725168 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:16.725173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:16.725180 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:16.725269 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:16.725699 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:16.725702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:16.725703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:16.725951 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:16.726121 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:16.726390 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 19:05:16.726558 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":115,"threads_started":1}
Jun 20 19:05:16 dist-test-slave-6wrf krb5kdc[5187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:16.739053 5271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:16.730150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45770 (local address 127.1.154.190:37893)
0620 19:05:16.730292 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:16.730302 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:16.730317 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:05:16.730415 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:16.730417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:16.730458 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:16.730532 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:16.731120 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.731672 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.732505 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:16.732852 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:16.732986 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:16.735532 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:16.735535 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:16.735537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:16.735570 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:16.737247 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:16.737836 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:16.737839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:16.737840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:16.737915 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:16.738177 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:16.738179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:16.738180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:16.738390 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:16.738630 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:16.738758 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 19:05:16.738854 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 19:05:16.739482 5229 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45770
I20260620 19:05:16.740190 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 5294
I20260620 19:05:16.745579 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 5428
I20260620 19:05:16.750806 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 5562
I20260620 19:05:16.756350 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 5200
2026-06-20T19:05:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2295 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:05:16 dist-test-slave-6wrf krb5kdc[5703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:05:16 dist-test-slave-6wrf krb5kdc[5703](info): set up 2 sockets
Jun 20 19:05:16 dist-test-slave-6wrf krb5kdc[5703](info): commencing operation
krb5kdc: starting...
W20260620 19:05:17.853894 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:05:17 dist-test-slave-6wrf krb5kdc[5703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:05:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:05:17.985392 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:32947
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:32947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:18.096446 5716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:18.096753 5716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:18.096833 5716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:18.101037 5716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:18.101118 5716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:18.101153 5716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:18.101208 5716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:18.101246 5716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:18.106824 5716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:32947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:32947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:18.107983 5716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:18.109032 5716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:18.118423 5721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:18.118572 5724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:18.118572 5722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:18.119028 5716 server_base.cc:1034] running on GCE node
I20260620 19:05:18.119603 5716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:18.120729 5716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:18.121959 5716 hybrid_clock.cc:648] HybridClock initialized: now 1781982318121949 us; error 39 us; skew 500 ppm
Jun 20 19:05:18 dist-test-slave-6wrf krb5kdc[5703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:18.124801 5716 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:05:18.125586 5716 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:18.125727 5716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:18.125943 5716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:18.127942 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "19ec578f321d4674b9a0888280fd2915"
format_stamp: "Formatted at 2026-06-20 19:05:18 on dist-test-slave-6wrf"
server_key: "d5da7fde060067bf3394c430a04bef41"
server_key_iv: "5881eb7c33757498b5d3309160156758"
server_key_version: "encryptionkey@0"
I20260620 19:05:18.128487 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "19ec578f321d4674b9a0888280fd2915"
format_stamp: "Formatted at 2026-06-20 19:05:18 on dist-test-slave-6wrf"
server_key: "d5da7fde060067bf3394c430a04bef41"
server_key_iv: "5881eb7c33757498b5d3309160156758"
server_key_version: "encryptionkey@0"
I20260620 19:05:18.133435 5716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 19:05:18.136018 5730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:18.137355 5716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:05:18.137513 5716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "19ec578f321d4674b9a0888280fd2915"
format_stamp: "Formatted at 2026-06-20 19:05:18 on dist-test-slave-6wrf"
server_key: "d5da7fde060067bf3394c430a04bef41"
server_key_iv: "5881eb7c33757498b5d3309160156758"
server_key_version: "encryptionkey@0"
I20260620 19:05:18.137669 5716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:18.150120 5716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:18.154072 5716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:18.154299 5716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:18.163456 5716 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:32947
I20260620 19:05:18.163738 5782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:32947 every 8 connection(s)
I20260620 19:05:18.164182 5716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:18.168155 5783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:18.172305 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 5716
I20260620 19:05:18.172416 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:18.172721 1642 external_mini_cluster.cc:1428] Setting key fff055f42c2a4d9519beee1a8a61c56b
I20260620 19:05:18.176095 5783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915: Bootstrap starting.
I20260620 19:05:18.178522 5783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:18.179315 5783 log.cc:824] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:18.181332 5783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915: No bootstrap required, opened a new log
Jun 20 19:05:18 dist-test-slave-6wrf krb5kdc[5703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:18.185367 5783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ec578f321d4674b9a0888280fd2915" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 32947 } }
I20260620 19:05:18.185643 5783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:18.185730 5783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ec578f321d4674b9a0888280fd2915, State: Initialized, Role: FOLLOWER
I20260620 19:05:18.186216 5783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ec578f321d4674b9a0888280fd2915" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 32947 } }
I20260620 19:05:18.186376 5783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:18.186439 5783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:18.186535 5783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:18.187597 5783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ec578f321d4674b9a0888280fd2915" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 32947 } }
I20260620 19:05:18.187964 5783 leader_election.cc:302] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19ec578f321d4674b9a0888280fd2915; no voters:
I20260620 19:05:18.188315 5783 leader_election.cc:288] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:18.188385 5788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:18.188572 5788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [term 1 LEADER]: Becoming Leader. State: Replica: 19ec578f321d4674b9a0888280fd2915, State: Running, Role: LEADER
I20260620 19:05:18.188833 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:18.174386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.1.154.190:32947)
0620 19:05:18.175098 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:18.175116 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:18.175169 (+ 53us) server_negotiation.cc:407] Connection header received
0620 19:05:18.175979 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:18.176001 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:18.176532 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:18.176992 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:18.178302 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.179369 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.180491 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.180961 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.181128 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:18.183895 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:18.183900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:18.183913 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:18.183938 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:18.185702 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:18.186286 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:18.186300 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:18.186305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:18.186458 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:18.186771 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:18.186773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:18.186774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:18.187166 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:18.187317 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:18.187623 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 19:05:18.187916 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":130,"threads_started":1}
I20260620 19:05:18.188863 5788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ec578f321d4674b9a0888280fd2915" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 32947 } }
I20260620 19:05:18.189862 5783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:18.190944 5788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19ec578f321d4674b9a0888280fd2915. Latest consensus state: current_term: 1 leader_uuid: "19ec578f321d4674b9a0888280fd2915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ec578f321d4674b9a0888280fd2915" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 32947 } } }
I20260620 19:05:18.191062 5788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:18.193126 5789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19ec578f321d4674b9a0888280fd2915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ec578f321d4674b9a0888280fd2915" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 32947 } } }
I20260620 19:05:18.193297 5789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915 [sys.catalog]: This master's current role is: LEADER
W20260620 19:05:18.194819 5803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:05:18.194877 5803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:05:18.194949 5797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:18.195596 5797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:18.201860 5797 catalog_manager.cc:1347] Generated new cluster ID: f89b0c975ae042e09a43ead216d1659e
I20260620 19:05:18.201970 5797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:18.221873 5797 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:18.222852 5797 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:18.234827 5797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915: Generated new TSK 0
I20260620 19:05:18.235558 5797 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:05:18.265946 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:32947
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:18.383249 5809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:18.383541 5809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:18.383589 5809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:18.388147 5809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:18.388218 5809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:18.388278 5809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:05:18.394624 5809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:32947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:18.395632 5809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:18.396570 5809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:18.405754 5815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:18.405773 5817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:18.405929 5809 server_base.cc:1034] running on GCE node
W20260620 19:05:18.406540 5814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:18.406929 5809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:18.407541 5809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:18.408799 5809 hybrid_clock.cc:648] HybridClock initialized: now 1781982318408768 us; error 30 us; skew 500 ppm
Jun 20 19:05:18 dist-test-slave-6wrf krb5kdc[5703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:18.412329 5809 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:05:18.413061 5809 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:18.413123 5809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:18.413281 5809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:18.415277 5809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1"
format_stamp: "Formatted at 2026-06-20 19:05:18 on dist-test-slave-6wrf"
server_key: "cbc41e8bb63df1fd5f1ac702c00baff7"
server_key_iv: "1e396a98a7c707c84cb7fde5faa677c5"
server_key_version: "encryptionkey@0"
I20260620 19:05:18.415767 5809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1"
format_stamp: "Formatted at 2026-06-20 19:05:18 on dist-test-slave-6wrf"
server_key: "cbc41e8bb63df1fd5f1ac702c00baff7"
server_key_iv: "1e396a98a7c707c84cb7fde5faa677c5"
server_key_version: "encryptionkey@0"
I20260620 19:05:18.419662 5809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:05:18.423087 5823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:18.424749 5809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:05:18.424891 5809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1"
format_stamp: "Formatted at 2026-06-20 19:05:18 on dist-test-slave-6wrf"
server_key: "cbc41e8bb63df1fd5f1ac702c00baff7"
server_key_iv: "1e396a98a7c707c84cb7fde5faa677c5"
server_key_version: "encryptionkey@0"
I20260620 19:05:18.424978 5809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:18.440312 5809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:18.443625 5809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:18.443775 5809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:18.444306 5809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:18.445158 5809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:18.445204 5809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:18.445243 5809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:18.445258 5809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:18.454257 5809 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:44631
I20260620 19:05:18.454495 5936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:44631 every 8 connection(s)
I20260620 19:05:18.454892 5809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:05:18.463614 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 5809
I20260620 19:05:18.463742 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:05:18.464018 1642 external_mini_cluster.cc:1428] Setting key e1ee34a19c17dbd77530ed28ea2185dd
Jun 20 19:05:18 dist-test-slave-6wrf krb5kdc[5703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:18.471700 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:18.457416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:34907 (local address 127.1.154.190:32947)
0620 19:05:18.457627 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:18.457631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:18.458708 (+ 1077us) server_negotiation.cc:407] Connection header received
0620 19:05:18.460088 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:18.460091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:18.460144 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:18.460256 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:18.461454 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.462060 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.463175 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.463485 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.463602 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:18.467018 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:18.467025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:18.467031 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:18.467064 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:18.469055 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:18.469817 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:18.469824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:18.469829 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:18.469931 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:18.470403 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:18.470407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:18.470409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:18.470652 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:18.470778 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:18.471407 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 19:05:18.471531 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 19:05:18.472502 5939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:18.457834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:32947 (local address 127.1.154.129:34907)
0620 19:05:18.458512 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:18.458571 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:18.459882 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:18.460423 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:18.460429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:18.460843 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:18.461313 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:18.461327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.462184 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.462189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:18.463052 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:18.463060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.463703 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.463709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:18.464116 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:18.464977 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:18.465002 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:18.466834 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:18.469216 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:18.469223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:18.469240 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:18.469679 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:18.470065 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:18.470071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:18.470076 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:18.470269 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:18.470767 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:18.470774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:18.471103 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 19:05:18.471355 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 19:05:18.471678 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":266,"threads_started":1}
I20260620 19:05:18.473753 5937 heartbeater.cc:347] Connected to a master server at 127.1.154.190:32947
I20260620 19:05:18.474035 5937 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:18.474598 5937 heartbeater.cc:511] Master 127.1.154.190:32947 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
I20260620 19:05:18.476600 5747 ts_manager.cc:194] Registered new tserver with Master: 5f27e6cfb3504ffe8e8f35f78bc57ce1 (127.1.154.129:44631)
I20260620 19:05:18.478325 5747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:34907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:05:18.496757 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:32947
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:18.617023 5942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:18.617479 5942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:18.617579 5942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:18.622974 5942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:18.623085 5942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:18.623234 5942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:18.630702 5942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:32947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:18.631879 5942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:18.632930 5942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:18.642835 5948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:18.642841 5947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:18.642875 5950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:18.643327 5942 server_base.cc:1034] running on GCE node
I20260620 19:05:18.643985 5942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:18.644603 5942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:18.645792 5942 hybrid_clock.cc:648] HybridClock initialized: now 1781982318645781 us; error 35 us; skew 500 ppm
Jun 20 19:05:18 dist-test-slave-6wrf krb5kdc[5703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:18.648556 5942 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:05:18.649269 5942 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:18.649338 5942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:18.649507 5942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:18.651515 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "1ba41e40d39642229322092b67e0f0cf"
format_stamp: "Formatted at 2026-06-20 19:05:18 on dist-test-slave-6wrf"
server_key: "52fb38376888a379eef6f9cc9cddab5e"
server_key_iv: "72397688582b59a75c4084a5845f88f2"
server_key_version: "encryptionkey@0"
I20260620 19:05:18.652010 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "1ba41e40d39642229322092b67e0f0cf"
format_stamp: "Formatted at 2026-06-20 19:05:18 on dist-test-slave-6wrf"
server_key: "52fb38376888a379eef6f9cc9cddab5e"
server_key_iv: "72397688582b59a75c4084a5845f88f2"
server_key_version: "encryptionkey@0"
I20260620 19:05:18.655948 5942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:05:18.659787 5956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:18.661473 5942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:18.661715 5942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "1ba41e40d39642229322092b67e0f0cf"
format_stamp: "Formatted at 2026-06-20 19:05:18 on dist-test-slave-6wrf"
server_key: "52fb38376888a379eef6f9cc9cddab5e"
server_key_iv: "72397688582b59a75c4084a5845f88f2"
server_key_version: "encryptionkey@0"
I20260620 19:05:18.661885 5942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:18.674240 5942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:18.677429 5942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:18.677697 5942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:18.678279 5942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:18.679085 5942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:18.679133 5942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:18.679173 5942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:18.679189 5942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:18.689213 5942 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:36159
I20260620 19:05:18.689277 6069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:36159 every 8 connection(s)
I20260620 19:05:18.689932 5942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:18.694651 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 5942
I20260620 19:05:18.694847 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:05:18.695385 1642 external_mini_cluster.cc:1428] Setting key 78d1121d42a28953c4dcd3e6b6f78174
Jun 20 19:05:18 dist-test-slave-6wrf krb5kdc[5703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
I20260620 19:05:18.705821 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:18.691796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:46363 (local address 127.1.154.190:32947)
0620 19:05:18.691928 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:18.691932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:18.692705 (+ 773us) server_negotiation.cc:407] Connection header received
0620 19:05:18.694125 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:18.694129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:18.694184 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:18.694288 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:18.695518 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.696157 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.697616 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.697957 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.698083 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:18.701371 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:18.701378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:18.701384 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:18.701416 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:18.703211 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:18.704047 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:18.704052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:18.704054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:18.704151 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:18.704587 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:18.704593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:18.704596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:18.704857 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:18.704965 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:18.705493 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 19:05:18.705650 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:05:18.706702 6072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:18.692133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:32947 (local address 127.1.154.130:46363)
0620 19:05:18.692550 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:18.692590 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:18.693894 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:18.694449 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:18.694456 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:18.694870 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:18.695379 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:18.695393 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.696294 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.696297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:18.697456 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:18.697465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.698132 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.698135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:18.698275 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:18.698908 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:18.698925 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:18.701030 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:18.703456 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:18.703466 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:18.703489 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:18.703908 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:18.704312 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:18.704315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:18.704317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:18.704476 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:18.704991 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:18.704997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:18.705231 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 19:05:18.705407 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 19:05:18.705856 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":102,"threads_started":1}
I20260620 19:05:18.708052 6070 heartbeater.cc:347] Connected to a master server at 127.1.154.190:32947
I20260620 19:05:18.708340 6070 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
I20260620 19:05:18.708869 6070 heartbeater.cc:511] Master 127.1.154.190:32947 requested a full tablet report, sending...
I20260620 19:05:18.710018 5747 ts_manager.cc:194] Registered new tserver with Master: 1ba41e40d39642229322092b67e0f0cf (127.1.154.130:36159)
I20260620 19:05:18.710860 5747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:46363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:05:18.725822 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:32947
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:18.839733 6075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:18.839987 6075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:18.840034 6075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:18.844732 6075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:18.844794 6075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:18.844858 6075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:18.850530 6075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:32947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:18.851661 6075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:18.852700 6075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:18.862275 6081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:18.862282 6080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:18.862409 6083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:18.862854 6075 server_base.cc:1034] running on GCE node
I20260620 19:05:18.863214 6075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:18.863785 6075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:18.864960 6075 hybrid_clock.cc:648] HybridClock initialized: now 1781982318864924 us; error 54 us; skew 500 ppm
Jun 20 19:05:18 dist-test-slave-6wrf krb5kdc[5703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:18.868257 6075 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:05:18.868968 6075 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:18.869059 6075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:18.869302 6075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:18.871585 6075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "97ec24aab6a849ec9d4f1784cba5e9fb"
format_stamp: "Formatted at 2026-06-20 19:05:18 on dist-test-slave-6wrf"
server_key: "23fdab5c341fc4c94c978531590731bb"
server_key_iv: "87c8e6577b42e66587f71f4af2d2e724"
server_key_version: "encryptionkey@0"
I20260620 19:05:18.872385 6075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "97ec24aab6a849ec9d4f1784cba5e9fb"
format_stamp: "Formatted at 2026-06-20 19:05:18 on dist-test-slave-6wrf"
server_key: "23fdab5c341fc4c94c978531590731bb"
server_key_iv: "87c8e6577b42e66587f71f4af2d2e724"
server_key_version: "encryptionkey@0"
I20260620 19:05:18.877985 6075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 19:05:18.881531 6089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:18.883173 6075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:18.883340 6075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "97ec24aab6a849ec9d4f1784cba5e9fb"
format_stamp: "Formatted at 2026-06-20 19:05:18 on dist-test-slave-6wrf"
server_key: "23fdab5c341fc4c94c978531590731bb"
server_key_iv: "87c8e6577b42e66587f71f4af2d2e724"
server_key_version: "encryptionkey@0"
I20260620 19:05:18.883482 6075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:18.905400 6075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:18.908974 6075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:18.909185 6075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:18.909924 6075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:18.910739 6075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:18.910789 6075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:18.910856 6075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:18.910873 6075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:18.920089 6075 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:34827
I20260620 19:05:18.920125 6202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:34827 every 8 connection(s)
I20260620 19:05:18.920679 6075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:18.923260 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 6075
I20260620 19:05:18.923401 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:05:18.923712 1642 external_mini_cluster.cc:1428] Setting key 09d781761e35eee366bdaf1b732d1b91
Jun 20 19:05:18 dist-test-slave-6wrf krb5kdc[5703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:18.939105 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:18.922526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:43037 (local address 127.1.154.190:32947)
0620 19:05:18.922672 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:18.922676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:18.924029 (+ 1353us) server_negotiation.cc:407] Connection header received
0620 19:05:18.926207 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:18.926210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:18.926268 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:18.926387 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:18.928316 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.929227 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.930489 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.930704 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.930788 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:18.934326 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:18.934329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:18.934332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:18.934355 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:18.935897 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:18.936556 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:18.936565 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:18.936570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:18.936758 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:18.937455 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:18.937462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:18.937466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:18.937738 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:18.937829 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:18.938777 (+ 948us) server_negotiation.cc:299] Negotiation successful
0620 19:05:18.938904 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:05:18.940449 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:18.922872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:32947 (local address 127.1.154.131:43037)
0620 19:05:18.923775 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:18.923871 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:18.925978 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:18.926680 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:18.926689 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:18.927200 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:18.928095 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:18.928115 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.929366 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.929368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:18.930359 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:18.930368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.930954 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.930958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:18.931182 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:18.932092 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:18.932114 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:18.934126 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:18.936028 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:18.936034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:18.936047 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:18.936306 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:18.936934 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:18.936939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:18.936942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:18.937296 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:18.937868 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:18.937880 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:18.938391 (+ 511us) client_negotiation.cc:769] Sending connection context
0620 19:05:18.938674 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 19:05:18.939279 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":153,"threads_started":1}
I20260620 19:05:18.941867 6203 heartbeater.cc:347] Connected to a master server at 127.1.154.190:32947
I20260620 19:05:18.942106 6203 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:18.942564 6203 heartbeater.cc:511] Master 127.1.154.190:32947 requested a full tablet report, sending...
I20260620 19:05:18.943622 5747 ts_manager.cc:194] Registered new tserver with Master: 97ec24aab6a849ec9d4f1784cba5e9fb (127.1.154.131:34827)
I20260620 19:05:18.944134 5747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:43037
I20260620 19:05:18.951990 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:05:18.963083 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:18.954238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.1.154.190:32947)
0620 19:05:18.954374 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:18.954379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:18.954615 (+ 236us) server_negotiation.cc:407] Connection header received
0620 19:05:18.954786 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:18.954789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:18.954840 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:18.954924 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:18.955653 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.956181 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.957012 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.957231 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.957339 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:18.959281 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:18.959289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:18.959292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:18.959340 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:18.961258 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:18.961797 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:18.961810 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:18.961811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:18.961888 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:18.962166 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:18.962171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:18.962173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:18.962407 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:18.962547 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:18.962790 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 19:05:18.962911 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:05:18.967093 5747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:05:18.970709 5747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:18.987399 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:18.981313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:44631 (local address 127.0.0.1:50670)
0620 19:05:18.982347 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:18.982360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:18.982433 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:18.983328 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:18.983331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:18.983350 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:18.983441 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:18.983447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.984784 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.984787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:18.985945 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:18.985951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.986719 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.986723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:18.986821 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:18.986856 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 19:05:18.986974 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 19:05:18.987099 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":58,"thread_start_us":103,"threads_started":1}
I20260620 19:05:18.987502 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:18.981893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.1.154.131:34827)
0620 19:05:18.982733 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:18.982744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:18.982763 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:05:18.982834 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:18.982839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:18.983004 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:18.983144 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:18.983353 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.984147 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.985881 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.986632 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.986768 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:18.986881 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:18.987247 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 19:05:18.987343 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":147,"threads_started":1}
I20260620 19:05:18.987769 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:18.981119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:36159 (local address 127.0.0.1:37016)
0620 19:05:18.981908 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:18.981960 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:18.982101 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:18.982694 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:18.982700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:18.982720 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:18.982869 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:18.982875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.984273 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.984279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:18.985695 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:18.985704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.986944 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.986948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:18.987073 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:18.987100 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:05:18.987169 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 19:05:18.987389 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":112,"thread_start_us":85,"threads_started":1}
I20260620 19:05:18.987836 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:18.981502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:34827 (local address 127.0.0.1:54612)
0620 19:05:18.982406 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:18.982421 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:18.982515 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:18.983133 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:18.983135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:18.983153 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:18.983237 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:18.983242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.984355 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.984358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:18.985766 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:18.985776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.986951 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.986956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:18.987086 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:18.987106 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:05:18.987171 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:05:18.987673 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":482,"thread_start_us":108,"threads_started":1}
I20260620 19:05:18.988114 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:18.981306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37016 (local address 127.1.154.130:36159)
0620 19:05:18.981801 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:18.981815 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:18.982238 (+ 423us) server_negotiation.cc:407] Connection header received
0620 19:05:18.982320 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:18.982325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:18.982544 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:18.982694 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:18.982980 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.984128 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.985809 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.986632 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.986752 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:18.986881 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:18.987836 (+ 955us) server_negotiation.cc:299] Negotiation successful
0620 19:05:18.987940 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":125,"threads_started":1}
I20260620 19:05:18.988345 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:18.981904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50670 (local address 127.1.154.129:44631)
0620 19:05:18.982733 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:18.982741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:18.982763 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:05:18.982834 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:18.982839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:18.983004 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:18.983144 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:18.983564 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.984640 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.986052 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:18.986597 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:18.986722 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:18.986881 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:18.987024 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 19:05:18.987129 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":89,"threads_started":1}
I20260620 19:05:18.991245 6137 tablet_service.cc:1467] Processing CreateTablet for tablet 8e22358ee65e47549a1308d919d725b2 (DEFAULT_TABLE table=test-table [id=2a634af36b704574a3a0b2165b8bcec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:18.992313 6137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e22358ee65e47549a1308d919d725b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:18.991823 5871 tablet_service.cc:1467] Processing CreateTablet for tablet 8e22358ee65e47549a1308d919d725b2 (DEFAULT_TABLE table=test-table [id=2a634af36b704574a3a0b2165b8bcec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:18.991823 6004 tablet_service.cc:1467] Processing CreateTablet for tablet 8e22358ee65e47549a1308d919d725b2 (DEFAULT_TABLE table=test-table [id=2a634af36b704574a3a0b2165b8bcec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:18.993098 5871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e22358ee65e47549a1308d919d725b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:18.993098 6004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e22358ee65e47549a1308d919d725b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:19.000684 6220 tablet_bootstrap.cc:492] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb: Bootstrap starting.
I20260620 19:05:19.002439 6222 tablet_bootstrap.cc:492] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf: Bootstrap starting.
I20260620 19:05:19.002894 6221 tablet_bootstrap.cc:492] T 8e22358ee65e47549a1308d919d725b2 P 5f27e6cfb3504ffe8e8f35f78bc57ce1: Bootstrap starting.
I20260620 19:05:19.003912 6220 tablet_bootstrap.cc:654] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:19.005342 6221 tablet_bootstrap.cc:654] T 8e22358ee65e47549a1308d919d725b2 P 5f27e6cfb3504ffe8e8f35f78bc57ce1: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:19.005354 6222 tablet_bootstrap.cc:654] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:19.005537 6220 log.cc:824] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:19.006119 6221 log.cc:824] T 8e22358ee65e47549a1308d919d725b2 P 5f27e6cfb3504ffe8e8f35f78bc57ce1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:19.006630 6222 log.cc:824] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:19.007903 6220 tablet_bootstrap.cc:492] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb: No bootstrap required, opened a new log
I20260620 19:05:19.008126 6220 ts_tablet_manager.cc:1397] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 19:05:19.008653 6222 tablet_bootstrap.cc:492] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf: No bootstrap required, opened a new log
I20260620 19:05:19.008831 6222 ts_tablet_manager.cc:1397] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 19:05:19.009470 6221 tablet_bootstrap.cc:492] T 8e22358ee65e47549a1308d919d725b2 P 5f27e6cfb3504ffe8e8f35f78bc57ce1: No bootstrap required, opened a new log
I20260620 19:05:19.009704 6221 ts_tablet_manager.cc:1397] T 8e22358ee65e47549a1308d919d725b2 P 5f27e6cfb3504ffe8e8f35f78bc57ce1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260620 19:05:19.012264 6220 raft_consensus.cc:348] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba41e40d39642229322092b67e0f0cf" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 36159 } } peers { permanent_uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44631 } } peers { permanent_uuid: "97ec24aab6a849ec9d4f1784cba5e9fb" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 34827 } }
I20260620 19:05:19.012567 6220 raft_consensus.cc:374] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:19.012621 6220 raft_consensus.cc:729] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ec24aab6a849ec9d4f1784cba5e9fb, State: Initialized, Role: FOLLOWER
I20260620 19:05:19.012540 6222 raft_consensus.cc:348] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba41e40d39642229322092b67e0f0cf" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 36159 } } peers { permanent_uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44631 } } peers { permanent_uuid: "97ec24aab6a849ec9d4f1784cba5e9fb" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 34827 } }
I20260620 19:05:19.012786 6222 raft_consensus.cc:374] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:19.012840 6222 raft_consensus.cc:729] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba41e40d39642229322092b67e0f0cf, State: Initialized, Role: FOLLOWER
I20260620 19:05:19.013103 6220 consensus_queue.cc:260] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba41e40d39642229322092b67e0f0cf" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 36159 } } peers { permanent_uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44631 } } peers { permanent_uuid: "97ec24aab6a849ec9d4f1784cba5e9fb" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 34827 } }
I20260620 19:05:19.013164 6222 consensus_queue.cc:260] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba41e40d39642229322092b67e0f0cf" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 36159 } } peers { permanent_uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44631 } } peers { permanent_uuid: "97ec24aab6a849ec9d4f1784cba5e9fb" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 34827 } }
I20260620 19:05:19.013545 6221 raft_consensus.cc:348] T 8e22358ee65e47549a1308d919d725b2 P 5f27e6cfb3504ffe8e8f35f78bc57ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba41e40d39642229322092b67e0f0cf" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 36159 } } peers { permanent_uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44631 } } peers { permanent_uuid: "97ec24aab6a849ec9d4f1784cba5e9fb" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 34827 } }
I20260620 19:05:19.013815 6221 raft_consensus.cc:374] T 8e22358ee65e47549a1308d919d725b2 P 5f27e6cfb3504ffe8e8f35f78bc57ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:19.013847 6203 heartbeater.cc:503] Master 127.1.154.190:32947 was elected leader, sending a full tablet report...
I20260620 19:05:19.013885 6221 raft_consensus.cc:729] T 8e22358ee65e47549a1308d919d725b2 P 5f27e6cfb3504ffe8e8f35f78bc57ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f27e6cfb3504ffe8e8f35f78bc57ce1, State: Initialized, Role: FOLLOWER
I20260620 19:05:19.014218 6070 heartbeater.cc:503] Master 127.1.154.190:32947 was elected leader, sending a full tablet report...
I20260620 19:05:19.014345 6221 consensus_queue.cc:260] T 8e22358ee65e47549a1308d919d725b2 P 5f27e6cfb3504ffe8e8f35f78bc57ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba41e40d39642229322092b67e0f0cf" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 36159 } } peers { permanent_uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44631 } } peers { permanent_uuid: "97ec24aab6a849ec9d4f1784cba5e9fb" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 34827 } }
I20260620 19:05:19.014942 5937 heartbeater.cc:503] Master 127.1.154.190:32947 was elected leader, sending a full tablet report...
I20260620 19:05:19.015122 6222 ts_tablet_manager.cc:1428] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:05:19.015229 6221 ts_tablet_manager.cc:1428] T 8e22358ee65e47549a1308d919d725b2 P 5f27e6cfb3504ffe8e8f35f78bc57ce1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:05:19.016119 6220 ts_tablet_manager.cc:1428] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 19:05:19.060388 6227 raft_consensus.cc:482] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:19.060748 6227 raft_consensus.cc:504] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba41e40d39642229322092b67e0f0cf" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 36159 } } peers { permanent_uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44631 } } peers { permanent_uuid: "97ec24aab6a849ec9d4f1784cba5e9fb" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 34827 } }
I20260620 19:05:19.062579 6227 leader_election.cc:288] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f27e6cfb3504ffe8e8f35f78bc57ce1 (127.1.154.129:44631), 97ec24aab6a849ec9d4f1784cba5e9fb (127.1.154.131:34827)
I20260620 19:05:19.066838 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:19.062824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:50059 (local address 127.1.154.129:44631)
0620 19:05:19.062985 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:19.063020 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:19.063226 (+ 206us) server_negotiation.cc:407] Connection header received
0620 19:05:19.063480 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:19.063484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:19.063561 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:19.063695 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:19.063989 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:19.064491 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:19.065703 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:19.066186 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:19.066329 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:19.066390 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:19.066596 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 19:05:19.066697 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:05:19.066823 6072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:19.062785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:44631 (local address 127.1.154.130:50059)
0620 19:05:19.063034 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:19.063052 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:19.063322 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:19.063711 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:19.063713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:19.063728 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:19.063872 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:19.063878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:19.064602 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:19.064608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:19.065523 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:19.065532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:19.066336 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:19.066339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:19.066462 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:19.066491 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 19:05:19.066566 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:05:19.066651 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":23}
I20260620 19:05:19.066953 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:19.063032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:40307 (local address 127.1.154.131:34827)
0620 19:05:19.063183 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:19.063190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:19.063662 (+ 472us) server_negotiation.cc:407] Connection header received
0620 19:05:19.063826 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:19.063829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:19.063886 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:19.063971 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:19.064243 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:19.064728 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:19.065867 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:19.066385 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:19.066510 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:19.066573 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:19.066764 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 19:05:19.066833 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:05:19.067138 6229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:19.062853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:34827 (local address 127.1.154.130:40307)
0620 19:05:19.063495 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:19.063511 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:19.063686 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:19.063989 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:19.063992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:19.064007 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:19.064130 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:19.064135 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:19.064849 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:19.064855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:19.065740 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:19.065750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:19.066503 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:19.066506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:19.066631 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:19.066652 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:05:19.066728 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:05:19.066792 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":23,"thread_start_us":164,"threads_started":1}
I20260620 19:05:19.067764 5891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e22358ee65e47549a1308d919d725b2" candidate_uuid: "1ba41e40d39642229322092b67e0f0cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1" is_pre_election: true
I20260620 19:05:19.067963 6157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e22358ee65e47549a1308d919d725b2" candidate_uuid: "1ba41e40d39642229322092b67e0f0cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ec24aab6a849ec9d4f1784cba5e9fb" is_pre_election: true
I20260620 19:05:19.068252 6157 raft_consensus.cc:2454] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ba41e40d39642229322092b67e0f0cf in term 0.
I20260620 19:05:19.068252 5891 raft_consensus.cc:2454] T 8e22358ee65e47549a1308d919d725b2 P 5f27e6cfb3504ffe8e8f35f78bc57ce1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ba41e40d39642229322092b67e0f0cf in term 0.
I20260620 19:05:19.068879 5959 leader_election.cc:302] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ba41e40d39642229322092b67e0f0cf, 97ec24aab6a849ec9d4f1784cba5e9fb; no voters:
I20260620 19:05:19.069245 6227 raft_consensus.cc:2781] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:19.069339 6227 raft_consensus.cc:482] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:19.069383 6227 raft_consensus.cc:3037] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:19.070571 6227 raft_consensus.cc:504] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba41e40d39642229322092b67e0f0cf" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 36159 } } peers { permanent_uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44631 } } peers { permanent_uuid: "97ec24aab6a849ec9d4f1784cba5e9fb" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 34827 } }
I20260620 19:05:19.070955 6227 leader_election.cc:288] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [CANDIDATE]: Term 1 election: Requested vote from peers 5f27e6cfb3504ffe8e8f35f78bc57ce1 (127.1.154.129:44631), 97ec24aab6a849ec9d4f1784cba5e9fb (127.1.154.131:34827)
I20260620 19:05:19.071352 5891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e22358ee65e47549a1308d919d725b2" candidate_uuid: "1ba41e40d39642229322092b67e0f0cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1"
I20260620 19:05:19.071364 6157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e22358ee65e47549a1308d919d725b2" candidate_uuid: "1ba41e40d39642229322092b67e0f0cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ec24aab6a849ec9d4f1784cba5e9fb"
I20260620 19:05:19.071532 5891 raft_consensus.cc:3037] T 8e22358ee65e47549a1308d919d725b2 P 5f27e6cfb3504ffe8e8f35f78bc57ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:19.071537 6157 raft_consensus.cc:3037] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:19.072592 6157 raft_consensus.cc:2454] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ba41e40d39642229322092b67e0f0cf in term 1.
I20260620 19:05:19.072597 5891 raft_consensus.cc:2454] T 8e22358ee65e47549a1308d919d725b2 P 5f27e6cfb3504ffe8e8f35f78bc57ce1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ba41e40d39642229322092b67e0f0cf in term 1.
I20260620 19:05:19.072942 5959 leader_election.cc:302] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ba41e40d39642229322092b67e0f0cf, 97ec24aab6a849ec9d4f1784cba5e9fb; no voters:
I20260620 19:05:19.073141 6227 raft_consensus.cc:2781] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:19.073421 6227 raft_consensus.cc:686] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [term 1 LEADER]: Becoming Leader. State: Replica: 1ba41e40d39642229322092b67e0f0cf, State: Running, Role: LEADER
I20260620 19:05:19.073820 6227 consensus_queue.cc:237] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba41e40d39642229322092b67e0f0cf" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 36159 } } peers { permanent_uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44631 } } peers { permanent_uuid: "97ec24aab6a849ec9d4f1784cba5e9fb" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 34827 } }
I20260620 19:05:19.077208 5747 catalog_manager.cc:5495] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ba41e40d39642229322092b67e0f0cf (127.1.154.130). New cstate: current_term: 1 leader_uuid: "1ba41e40d39642229322092b67e0f0cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba41e40d39642229322092b67e0f0cf" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 36159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ec24aab6a849ec9d4f1784cba5e9fb" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 34827 } health_report { overall_health: UNKNOWN } } }
I20260620 19:05:19.089848 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:19.085683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.1.154.130:36159)
0620 19:05:19.085825 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:19.085829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:19.085938 (+ 109us) server_negotiation.cc:407] Connection header received
0620 19:05:19.086257 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:19.086260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:19.086318 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:19.086413 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:19.086747 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:19.087362 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:19.088263 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:19.088478 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:19.088588 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:19.088895 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:19.089332 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:19.089467 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:19.089527 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 19:05:19.089579 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:05:19.095882 6157 raft_consensus.cc:1261] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb [term 1 FOLLOWER]: Refusing update from remote peer 1ba41e40d39642229322092b67e0f0cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:19.095885 5891 raft_consensus.cc:1261] T 8e22358ee65e47549a1308d919d725b2 P 5f27e6cfb3504ffe8e8f35f78bc57ce1 [term 1 FOLLOWER]: Refusing update from remote peer 1ba41e40d39642229322092b67e0f0cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:19.096581 6230 consensus_queue.cc:1035] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ec24aab6a849ec9d4f1784cba5e9fb" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 34827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:19.096758 6227 consensus_queue.cc:1035] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f27e6cfb3504ffe8e8f35f78bc57ce1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:19.112527 5745 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:19.112730 5745 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:19.115190 5745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19ec578f321d4674b9a0888280fd2915: Sending DeleteTablet for 3 replicas of tablet 8e22358ee65e47549a1308d919d725b2
I20260620 19:05:19.116209 6137 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e22358ee65e47549a1308d919d725b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:19 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:54612
I20260620 19:05:19.116461 5871 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e22358ee65e47549a1308d919d725b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:19 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:50670
I20260620 19:05:19.116565 6004 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e22358ee65e47549a1308d919d725b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:19 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:37016
I20260620 19:05:19.117043 6243 tablet_replica.cc:330] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb: stopping tablet replica
I20260620 19:05:19.117395 6243 raft_consensus.cc:2229] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:19.117396 6245 tablet_replica.cc:330] T 8e22358ee65e47549a1308d919d725b2 P 5f27e6cfb3504ffe8e8f35f78bc57ce1: stopping tablet replica
I20260620 19:05:19.117753 6243 raft_consensus.cc:2258] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:19.117772 6244 tablet_replica.cc:330] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf: stopping tablet replica
I20260620 19:05:19.117861 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 5809
I20260620 19:05:19.118609 6244 raft_consensus.cc:2229] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:05:19.118947 6244 raft_consensus.cc:2258] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:19.120268 6243 ts_tablet_manager.cc:1905] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:19.123378 6243 ts_tablet_manager.cc:1918] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:19.123531 6243 log.cc:1196] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/8e22358ee65e47549a1308d919d725b2
I20260620 19:05:19.123929 6243 ts_tablet_manager.cc:1939] T 8e22358ee65e47549a1308d919d725b2 P 97ec24aab6a849ec9d4f1784cba5e9fb: Deleting consensus metadata
I20260620 19:05:19.123993 6244 ts_tablet_manager.cc:1905] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:19.125362 5733 catalog_manager.cc:4841] TS 97ec24aab6a849ec9d4f1784cba5e9fb (127.1.154.131:34827): tablet 8e22358ee65e47549a1308d919d725b2 (table test-table [id=2a634af36b704574a3a0b2165b8bcec9]) successfully deleted
W20260620 19:05:19.126060 5731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:44631 (error 108)
I20260620 19:05:19.126631 6244 ts_tablet_manager.cc:1918] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:19.126739 6244 log.cc:1196] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/8e22358ee65e47549a1308d919d725b2
I20260620 19:05:19.127022 6244 ts_tablet_manager.cc:1939] T 8e22358ee65e47549a1308d919d725b2 P 1ba41e40d39642229322092b67e0f0cf: Deleting consensus metadata
I20260620 19:05:19.127127 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:19.126799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:44631 (local address 127.0.0.1:50676)
0620 19:05:19.126979 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:19.127047 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:44631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260620 19:05:19.127282 5731 catalog_manager.cc:4584] TS 5f27e6cfb3504ffe8e8f35f78bc57ce1 (127.1.154.129:44631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e22358ee65e47549a1308d919d725b2: Network error: Client connection negotiation failed: client connection to 127.1.154.129:44631: connect: Connection refused (error 111)
I20260620 19:05:19.127825 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 5942
I20260620 19:05:19.133934 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 6075
I20260620 19:05:19.137019 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:19.136431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:36159 (local address 127.0.0.1:37028)
0620 19:05:19.136661 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:19.136777 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.130:36159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 19:05:19.140594 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 5716
2026-06-20T19:05:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2385 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T19:05:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:19Z Disabled control of system clock
I20260620 19:05:19.172171 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:39377
--webserver_interface=127.1.154.190
--webserver_port=0
--builtin_ntp_servers=127.1.154.148:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:39377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:05:19.283385 6250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:19.283658 6250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:19.283694 6250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:19.287839 6250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:19.287896 6250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:19.287911 6250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:19.287928 6250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:19.287943 6250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:19.292742 6250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:39377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:39377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:19.294082 6250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:19.295375 6250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:19.305682 6256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:19.305691 6255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:19.305691 6258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:19.305979 6250 server_base.cc:1034] running on GCE node
I20260620 19:05:19.306561 6250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:19.307641 6250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:19.309063 6250 hybrid_clock.cc:648] HybridClock initialized: now 1781982319309013 us; error 29 us; skew 500 ppm
I20260620 19:05:19.311368 6250 webserver.cc:460] Webserver started at http://127.1.154.190:42271/ using document root <none> and password file <none>
I20260620 19:05:19.311967 6250 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:19.312031 6250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:19.312249 6250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:19.314349 6250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "3d1ab3fd55e44d1ead90561b37b409c5"
format_stamp: "Formatted at 2026-06-20 19:05:19 on dist-test-slave-6wrf"
server_key: "3ee452c86b416d9d6724f65d54bf22a9"
server_key_iv: "afcb393723bd8b8ab035e9eb19fa08b9"
server_key_version: "encryptionkey@0"
I20260620 19:05:19.314901 6250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "3d1ab3fd55e44d1ead90561b37b409c5"
format_stamp: "Formatted at 2026-06-20 19:05:19 on dist-test-slave-6wrf"
server_key: "3ee452c86b416d9d6724f65d54bf22a9"
server_key_iv: "afcb393723bd8b8ab035e9eb19fa08b9"
server_key_version: "encryptionkey@0"
I20260620 19:05:19.318868 6250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:05:19.321447 6264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:19.322741 6250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:05:19.322901 6250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "3d1ab3fd55e44d1ead90561b37b409c5"
format_stamp: "Formatted at 2026-06-20 19:05:19 on dist-test-slave-6wrf"
server_key: "3ee452c86b416d9d6724f65d54bf22a9"
server_key_iv: "afcb393723bd8b8ab035e9eb19fa08b9"
server_key_version: "encryptionkey@0"
I20260620 19:05:19.323020 6250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:19.327688 6250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:19.328361 6250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:19.328549 6250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:19.335783 6250 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:39377
I20260620 19:05:19.335788 6316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:39377 every 8 connection(s)
I20260620 19:05:19.336706 6250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:19.338598 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 6250
I20260620 19:05:19.338778 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:19.339156 1642 external_mini_cluster.cc:1428] Setting key 14ce78e2416b47b74d0edc777e950883
I20260620 19:05:19.341871 6317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:19.346323 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:19.341274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48466 (local address 127.1.154.190:39377)
0620 19:05:19.342314 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:19.342332 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:19.342369 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:05:19.343415 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:19.343438 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:19.343815 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:19.344157 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:19.344992 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:19.344996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:19.345007 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:19.345017 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:19.345068 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:19.345084 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:19.345249 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:19.345318 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 19:05:19.345547 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":156,"threads_started":1}
I20260620 19:05:19.348729 6317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5: Bootstrap starting.
I20260620 19:05:19.351207 6317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:19.352094 6317 log.cc:824] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:19.354707 6317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5: No bootstrap required, opened a new log
I20260620 19:05:19.358683 6317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1ab3fd55e44d1ead90561b37b409c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39377 } }
I20260620 19:05:19.358964 6317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:19.359062 6317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1ab3fd55e44d1ead90561b37b409c5, State: Initialized, Role: FOLLOWER
I20260620 19:05:19.359634 6317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1ab3fd55e44d1ead90561b37b409c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39377 } }
I20260620 19:05:19.359813 6317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:19.359887 6317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:19.359999 6317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:19.360981 6317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1ab3fd55e44d1ead90561b37b409c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39377 } }
I20260620 19:05:19.361353 6317 leader_election.cc:302] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d1ab3fd55e44d1ead90561b37b409c5; no voters:
I20260620 19:05:19.361799 6317 leader_election.cc:288] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:19.361941 6322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:19.362195 6322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1ab3fd55e44d1ead90561b37b409c5, State: Running, Role: LEADER
I20260620 19:05:19.362505 6322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1ab3fd55e44d1ead90561b37b409c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39377 } }
I20260620 19:05:19.363091 6317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:19.364151 6323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d1ab3fd55e44d1ead90561b37b409c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1ab3fd55e44d1ead90561b37b409c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39377 } } }
I20260620 19:05:19.364137 6324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d1ab3fd55e44d1ead90561b37b409c5. Latest consensus state: current_term: 1 leader_uuid: "3d1ab3fd55e44d1ead90561b37b409c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1ab3fd55e44d1ead90561b37b409c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39377 } } }
I20260620 19:05:19.364289 6323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:19.364289 6324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:19.364755 6331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:19.367986 6331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:19.373960 6331 catalog_manager.cc:1347] Generated new cluster ID: b6be982033364ecabfae3e8a0a4eb085
I20260620 19:05:19.374047 6331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:19.385033 6331 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:19.385383 6331 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:19.396358 6331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5: Generated new TSK 0
I20260620 19:05:19.397073 6331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:05:19.403602 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39377
--builtin_ntp_servers=127.1.154.148:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:05:19.526460 6341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:19.526727 6341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:19.526777 6341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:19.531505 6341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:19.531566 6341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:19.531625 6341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:05:19.536950 6341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:19.538100 6341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:19.539469 6341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:05:19.549623 6341 server_base.cc:1034] running on GCE node
W20260620 19:05:19.549670 6349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:19.549760 6347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:19.549561 6346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:19.550309 6341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:19.551090 6341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:19.552289 6341 hybrid_clock.cc:648] HybridClock initialized: now 1781982319552262 us; error 52 us; skew 500 ppm
I20260620 19:05:19.554311 6341 webserver.cc:460] Webserver started at http://127.1.154.129:45139/ using document root <none> and password file <none>
I20260620 19:05:19.554963 6341 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:19.555058 6341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:19.555286 6341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:19.557313 6341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "2ebb7862b568434d90432d01ee651bd4"
format_stamp: "Formatted at 2026-06-20 19:05:19 on dist-test-slave-6wrf"
server_key: "660aba37c21cf0b4ffea14c7abd5a6a3"
server_key_iv: "666fddfc67a302cf45cc38728e5b2bce"
server_key_version: "encryptionkey@0"
I20260620 19:05:19.557893 6341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "2ebb7862b568434d90432d01ee651bd4"
format_stamp: "Formatted at 2026-06-20 19:05:19 on dist-test-slave-6wrf"
server_key: "660aba37c21cf0b4ffea14c7abd5a6a3"
server_key_iv: "666fddfc67a302cf45cc38728e5b2bce"
server_key_version: "encryptionkey@0"
I20260620 19:05:19.561594 6341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:05:19.564167 6355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:19.565425 6341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:05:19.565559 6341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "2ebb7862b568434d90432d01ee651bd4"
format_stamp: "Formatted at 2026-06-20 19:05:19 on dist-test-slave-6wrf"
server_key: "660aba37c21cf0b4ffea14c7abd5a6a3"
server_key_iv: "666fddfc67a302cf45cc38728e5b2bce"
server_key_version: "encryptionkey@0"
I20260620 19:05:19.565743 6341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:19.570288 6341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:19.570873 6341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:19.571060 6341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:19.571599 6341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:19.572487 6341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:19.572561 6341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:19.572634 6341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:19.572678 6341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:19.582150 6341 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:45561
I20260620 19:05:19.582221 6468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:45561 every 8 connection(s)
I20260620 19:05:19.583106 6341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:05:19.590642 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:19.584999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:42347 (local address 127.1.154.190:39377)
0620 19:05:19.585272 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:19.585277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:19.586274 (+ 997us) server_negotiation.cc:407] Connection header received
0620 19:05:19.587563 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:19.587566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:19.587605 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:19.587717 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:19.590123 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:19.590126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:19.590128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:19.590141 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:19.590162 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:19.590169 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:19.590262 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:19.590487 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 19:05:19.590530 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260620 19:05:19.591048 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 6341
I20260620 19:05:19.591184 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:05:19.591499 1642 external_mini_cluster.cc:1428] Setting key 4c20901de836da9ed5c03eed81ff8c89
I20260620 19:05:19.591838 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:19.585403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39377 (local address 127.1.154.129:42347)
0620 19:05:19.586110 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:19.586150 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:19.587303 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:19.588076 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:19.588096 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:19.588241 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:19.589589 (+ 1348us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:05:19.589591 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:19.589721 (+ 130us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:05:19.589836 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:19.590323 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:19.590332 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:19.590337 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:05:19.590507 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 19:05:19.590804 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":129,"threads_started":1}
I20260620 19:05:19.593145 6469 heartbeater.cc:347] Connected to a master server at 127.1.154.190:39377
I20260620 19:05:19.593387 6469 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:19.593724 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39377
--builtin_ntp_servers=127.1.154.148:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 19:05:19.593940 6469 heartbeater.cc:511] Master 127.1.154.190:39377 requested a full tablet report, sending...
I20260620 19:05:19.595557 6281 ts_manager.cc:194] Registered new tserver with Master: 2ebb7862b568434d90432d01ee651bd4 (127.1.154.129:45561)
W20260620 19:05:19.719182 6472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:19.719484 6472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:19.719527 6472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:19.723562 6472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:19.723624 6472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:19.723680 6472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:19.728627 6472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:19.729758 6472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:19.731127 6472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:19.741531 6477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:19.741473 6480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:19.741565 6472 server_base.cc:1034] running on GCE node
W20260620 19:05:19.741494 6478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:19.742550 6472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:19.743491 6472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:19.744748 6472 hybrid_clock.cc:648] HybridClock initialized: now 1781982319744697 us; error 66 us; skew 500 ppm
I20260620 19:05:19.747140 6472 webserver.cc:460] Webserver started at http://127.1.154.130:43491/ using document root <none> and password file <none>
I20260620 19:05:19.747826 6472 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:19.747931 6472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:19.748158 6472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:19.750615 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "420b44f531184026b7016ea42a45063e"
format_stamp: "Formatted at 2026-06-20 19:05:19 on dist-test-slave-6wrf"
server_key: "616eb22f9a1d2ed0f84efbc8d6466bbe"
server_key_iv: "48556ca5554763220fb8276e197c9013"
server_key_version: "encryptionkey@0"
I20260620 19:05:19.751273 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "420b44f531184026b7016ea42a45063e"
format_stamp: "Formatted at 2026-06-20 19:05:19 on dist-test-slave-6wrf"
server_key: "616eb22f9a1d2ed0f84efbc8d6466bbe"
server_key_iv: "48556ca5554763220fb8276e197c9013"
server_key_version: "encryptionkey@0"
I20260620 19:05:19.755545 6472 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:05:19.758531 6486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:19.760377 6472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:19.760603 6472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "420b44f531184026b7016ea42a45063e"
format_stamp: "Formatted at 2026-06-20 19:05:19 on dist-test-slave-6wrf"
server_key: "616eb22f9a1d2ed0f84efbc8d6466bbe"
server_key_iv: "48556ca5554763220fb8276e197c9013"
server_key_version: "encryptionkey@0"
I20260620 19:05:19.760811 6472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:19.767768 6472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:19.768747 6472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:19.768986 6472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:19.769838 6472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:19.771449 6472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:19.771560 6472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:19.771649 6472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:19.771701 6472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:19.783238 6472 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:38329
I20260620 19:05:19.783258 6599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:38329 every 8 connection(s)
I20260620 19:05:19.784696 6472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:19.791471 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 6472
I20260620 19:05:19.791654 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:05:19.791985 1642 external_mini_cluster.cc:1428] Setting key 4b449805b03704fad264d1e2fc6c4194
I20260620 19:05:19.792847 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:19.786940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:44045 (local address 127.1.154.190:39377)
0620 19:05:19.787161 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:19.787167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:19.788112 (+ 945us) server_negotiation.cc:407] Connection header received
0620 19:05:19.789920 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:19.789923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:19.789962 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:19.790053 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:19.792055 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:19.792059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:19.792063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:19.792100 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:19.792128 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:19.792141 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:19.792266 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:19.792637 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 19:05:19.792679 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 19:05:19.794394 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39377
--builtin_ntp_servers=127.1.154.148:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 19:05:19.794561 6602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:19.787181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39377 (local address 127.1.154.130:44045)
0620 19:05:19.787914 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:19.787957 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:19.789467 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:19.790563 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:19.790586 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:19.790692 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:19.791574 (+ 882us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:05:19.791576 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:19.791686 (+ 110us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:05:19.791856 (+ 170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:19.792335 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:19.792344 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:19.792349 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:05:19.792624 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 19:05:19.793006 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":108,"threads_started":1}
I20260620 19:05:19.796151 6600 heartbeater.cc:347] Connected to a master server at 127.1.154.190:39377
I20260620 19:05:19.796525 6600 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:19.797086 6600 heartbeater.cc:511] Master 127.1.154.190:39377 requested a full tablet report, sending...
I20260620 19:05:19.798306 6281 ts_manager.cc:194] Registered new tserver with Master: 420b44f531184026b7016ea42a45063e (127.1.154.130:38329)
W20260620 19:05:19.911271 6603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:19.911543 6603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:19.911594 6603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:19.915699 6603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:19.915762 6603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:19.915822 6603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:19.920610 6603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:19.921779 6603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:19.923151 6603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:19.934557 6608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:19.934739 6611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:19.934664 6609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:19.934906 6603 server_base.cc:1034] running on GCE node
I20260620 19:05:19.935411 6603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:19.936203 6603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:19.937439 6603 hybrid_clock.cc:648] HybridClock initialized: now 1781982319937422 us; error 30 us; skew 500 ppm
I20260620 19:05:19.939495 6603 webserver.cc:460] Webserver started at http://127.1.154.131:41131/ using document root <none> and password file <none>
I20260620 19:05:19.940109 6603 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:19.940207 6603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:19.940445 6603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:19.942528 6603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "98b1a849d5b4456c9e2972565f5c2268"
format_stamp: "Formatted at 2026-06-20 19:05:19 on dist-test-slave-6wrf"
server_key: "feb580bf05a0dc4d202e6d484f016dcf"
server_key_iv: "4c7de447e899c2d18edd6fa9b8741862"
server_key_version: "encryptionkey@0"
I20260620 19:05:19.943076 6603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "98b1a849d5b4456c9e2972565f5c2268"
format_stamp: "Formatted at 2026-06-20 19:05:19 on dist-test-slave-6wrf"
server_key: "feb580bf05a0dc4d202e6d484f016dcf"
server_key_iv: "4c7de447e899c2d18edd6fa9b8741862"
server_key_version: "encryptionkey@0"
I20260620 19:05:19.948357 6603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 19:05:19.951113 6617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:19.952718 6603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:05:19.952884 6603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "98b1a849d5b4456c9e2972565f5c2268"
format_stamp: "Formatted at 2026-06-20 19:05:19 on dist-test-slave-6wrf"
server_key: "feb580bf05a0dc4d202e6d484f016dcf"
server_key_iv: "4c7de447e899c2d18edd6fa9b8741862"
server_key_version: "encryptionkey@0"
I20260620 19:05:19.953013 6603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:19.957702 6603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:19.958293 6603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:19.958477 6603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:19.959024 6603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:19.959928 6603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:19.960000 6603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:19.960062 6603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:19.960114 6603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:19.970263 6603 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:33993
I20260620 19:05:19.970280 6730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:33993 every 8 connection(s)
I20260620 19:05:19.971247 6603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:19.971570 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 6603
I20260620 19:05:19.971705 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:05:19.972026 1642 external_mini_cluster.cc:1428] Setting key d49faa952f8af6670a044762652b47e5
I20260620 19:05:19.977491 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:19.972999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:38001 (local address 127.1.154.190:39377)
0620 19:05:19.973131 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:19.973136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:19.974371 (+ 1235us) server_negotiation.cc:407] Connection header received
0620 19:05:19.975604 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:19.975607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:19.975652 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:19.975728 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:19.976934 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:19.976938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:19.976941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:19.976962 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:19.976996 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:19.977006 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:19.977140 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:19.977347 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 19:05:19.977389 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:05:19.978278 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:19.973451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39377 (local address 127.1.154.131:38001)
0620 19:05:19.974200 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:19.974248 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:19.975407 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:19.975880 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:19.975886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:19.975971 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:19.976581 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:05:19.976582 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:19.976622 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:05:19.976685 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:19.977145 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:19.977152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:19.977156 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:05:19.977295 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 19:05:19.977471 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":197,"threads_started":1}
I20260620 19:05:19.979342 6731 heartbeater.cc:347] Connected to a master server at 127.1.154.190:39377
I20260620 19:05:19.979532 6731 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:19.979861 6731 heartbeater.cc:511] Master 127.1.154.190:39377 requested a full tablet report, sending...
I20260620 19:05:19.980785 6281 ts_manager.cc:194] Registered new tserver with Master: 98b1a849d5b4456c9e2972565f5c2268 (127.1.154.131:33993)
I20260620 19:05:19.985711 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:05:19.989408 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:19.987665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48472 (local address 127.1.154.190:39377)
0620 19:05:19.987848 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:19.987852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:19.988001 (+ 149us) server_negotiation.cc:407] Connection header received
0620 19:05:19.988145 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:19.988148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:19.988187 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:19.988284 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:19.988978 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:19.988981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:19.988984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:19.988995 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:19.989015 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:19.989024 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:19.989124 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:19.989241 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 19:05:19.989282 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 19:05:19.992506 6281 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:05:19.995570 6281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:20.007718 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.004678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.1.154.130:38329)
0620 19:05:20.005150 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:20.005164 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:20.005566 (+ 402us) server_negotiation.cc:407] Connection header received
0620 19:05:20.005762 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:20.005769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:20.005966 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:20.006106 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:20.007149 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:20.007155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:20.007166 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:20.007178 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:20.007222 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:20.007241 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:20.007351 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:20.007463 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 19:05:20.007531 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":130,"threads_started":1}
I20260620 19:05:20.008203 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.004325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:38329 (local address 127.0.0.1:54428)
0620 19:05:20.005433 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:20.005479 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:20.005648 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:20.006112 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:20.006118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:20.006289 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:20.006926 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:05:20.006928 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:20.006948 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:05:20.007014 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:20.007336 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:20.007343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:20.007348 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:05:20.007453 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 19:05:20.007518 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":71,"threads_started":1}
I20260620 19:05:20.008383 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.004523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:33993 (local address 127.0.0.1:59364)
0620 19:05:20.005433 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:20.005480 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:20.005574 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:20.006916 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:20.006922 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:20.007168 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:20.007586 (+ 418us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:05:20.007588 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:20.007597 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:05:20.007635 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:20.008027 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:20.008030 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:20.008033 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:05:20.008116 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 19:05:20.008149 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":100,"threads_started":1}
I20260620 19:05:20.008502 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.005040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.1.154.131:33993)
0620 19:05:20.005674 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:20.005687 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:20.005710 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:05:20.005779 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:20.005783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:20.005967 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:20.006097 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:20.007774 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:20.007782 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:20.007790 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:20.007875 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:20.007912 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:20.007936 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:20.008031 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:20.008268 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 19:05:20.008336 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":151,"threads_started":1}
I20260620 19:05:20.008822 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.005057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47628 (local address 127.1.154.129:45561)
0620 19:05:20.005475 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:20.005486 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:20.005751 (+ 265us) server_negotiation.cc:407] Connection header received
0620 19:05:20.005908 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:20.005913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:20.006051 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:20.006159 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:20.007520 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:20.007528 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:20.007545 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:20.007562 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:20.007600 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:20.007628 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:20.007711 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:20.008168 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 19:05:20.008250 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":114,"threads_started":1}
I20260620 19:05:20.009122 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.004724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:45561 (local address 127.0.0.1:47628)
0620 19:05:20.005670 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:20.005684 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:20.005798 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:20.006755 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:20.006760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:20.006778 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:20.007337 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:05:20.007339 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:20.007349 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:05:20.007376 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:20.008036 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:20.008041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:20.008048 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 19:05:20.008983 (+ 935us) client_negotiation.cc:240] Negotiation successful
0620 19:05:20.009017 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":84,"threads_started":1}
I20260620 19:05:20.010885 6662 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3e24fe69ce4dda92d4e4c4b9da23a3 (DEFAULT_TABLE table=test-table [id=29571b0da5c74214a010524fd7633d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:20.010955 6534 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3e24fe69ce4dda92d4e4c4b9da23a3 (DEFAULT_TABLE table=test-table [id=29571b0da5c74214a010524fd7633d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:20.012022 6662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3e24fe69ce4dda92d4e4c4b9da23a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:20.012025 6534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3e24fe69ce4dda92d4e4c4b9da23a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:20.013873 6403 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3e24fe69ce4dda92d4e4c4b9da23a3 (DEFAULT_TABLE table=test-table [id=29571b0da5c74214a010524fd7633d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:20.014875 6403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3e24fe69ce4dda92d4e4c4b9da23a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:20.018015 6748 tablet_bootstrap.cc:492] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268: Bootstrap starting.
I20260620 19:05:20.019377 6749 tablet_bootstrap.cc:492] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e: Bootstrap starting.
I20260620 19:05:20.020390 6748 tablet_bootstrap.cc:654] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:20.021221 6748 log.cc:824] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:20.021351 6749 tablet_bootstrap.cc:654] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:20.021494 6750 tablet_bootstrap.cc:492] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4: Bootstrap starting.
I20260620 19:05:20.022125 6749 log.cc:824] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:20.023386 6750 tablet_bootstrap.cc:654] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:20.023667 6748 tablet_bootstrap.cc:492] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268: No bootstrap required, opened a new log
I20260620 19:05:20.023861 6748 ts_tablet_manager.cc:1397] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:05:20.024099 6750 log.cc:824] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:20.024283 6749 tablet_bootstrap.cc:492] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e: No bootstrap required, opened a new log
I20260620 19:05:20.024451 6749 ts_tablet_manager.cc:1397] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 19:05:20.025856 6750 tablet_bootstrap.cc:492] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4: No bootstrap required, opened a new log
I20260620 19:05:20.026063 6750 ts_tablet_manager.cc:1397] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:05:20.029364 6750 raft_consensus.cc:348] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "420b44f531184026b7016ea42a45063e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38329 } } peers { permanent_uuid: "98b1a849d5b4456c9e2972565f5c2268" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33993 } } peers { permanent_uuid: "2ebb7862b568434d90432d01ee651bd4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45561 } }
I20260620 19:05:20.029353 6748 raft_consensus.cc:348] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "420b44f531184026b7016ea42a45063e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38329 } } peers { permanent_uuid: "98b1a849d5b4456c9e2972565f5c2268" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33993 } } peers { permanent_uuid: "2ebb7862b568434d90432d01ee651bd4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45561 } }
I20260620 19:05:20.029604 6750 raft_consensus.cc:374] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:20.029659 6748 raft_consensus.cc:374] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:20.029697 6750 raft_consensus.cc:729] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebb7862b568434d90432d01ee651bd4, State: Initialized, Role: FOLLOWER
I20260620 19:05:20.029723 6748 raft_consensus.cc:729] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98b1a849d5b4456c9e2972565f5c2268, State: Initialized, Role: FOLLOWER
I20260620 19:05:20.029866 6749 raft_consensus.cc:348] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "420b44f531184026b7016ea42a45063e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38329 } } peers { permanent_uuid: "98b1a849d5b4456c9e2972565f5c2268" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33993 } } peers { permanent_uuid: "2ebb7862b568434d90432d01ee651bd4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45561 } }
I20260620 19:05:20.030117 6749 raft_consensus.cc:374] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:20.030185 6749 raft_consensus.cc:729] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420b44f531184026b7016ea42a45063e, State: Initialized, Role: FOLLOWER
I20260620 19:05:20.030160 6750 consensus_queue.cc:260] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "420b44f531184026b7016ea42a45063e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38329 } } peers { permanent_uuid: "98b1a849d5b4456c9e2972565f5c2268" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33993 } } peers { permanent_uuid: "2ebb7862b568434d90432d01ee651bd4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45561 } }
I20260620 19:05:20.030205 6748 consensus_queue.cc:260] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "420b44f531184026b7016ea42a45063e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38329 } } peers { permanent_uuid: "98b1a849d5b4456c9e2972565f5c2268" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33993 } } peers { permanent_uuid: "2ebb7862b568434d90432d01ee651bd4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45561 } }
I20260620 19:05:20.030633 6749 consensus_queue.cc:260] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "420b44f531184026b7016ea42a45063e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38329 } } peers { permanent_uuid: "98b1a849d5b4456c9e2972565f5c2268" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33993 } } peers { permanent_uuid: "2ebb7862b568434d90432d01ee651bd4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45561 } }
I20260620 19:05:20.030824 6469 heartbeater.cc:503] Master 127.1.154.190:39377 was elected leader, sending a full tablet report...
I20260620 19:05:20.031188 6750 ts_tablet_manager.cc:1428] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260620 19:05:20.031219 6748 ts_tablet_manager.cc:1428] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 19:05:20.031538 6731 heartbeater.cc:503] Master 127.1.154.190:39377 was elected leader, sending a full tablet report...
I20260620 19:05:20.031864 6600 heartbeater.cc:503] Master 127.1.154.190:39377 was elected leader, sending a full tablet report...
I20260620 19:05:20.032146 6749 ts_tablet_manager.cc:1428] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e: Time spent starting tablet: real 0.008s user 0.008s sys 0.001s
W20260620 19:05:20.036021 6601 tablet.cc:2369] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:05:20.084353 6470 tablet.cc:2369] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:05:20.121330 6754 raft_consensus.cc:482] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:20.121655 6754 raft_consensus.cc:504] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "420b44f531184026b7016ea42a45063e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38329 } } peers { permanent_uuid: "98b1a849d5b4456c9e2972565f5c2268" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33993 } } peers { permanent_uuid: "2ebb7862b568434d90432d01ee651bd4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45561 } }
I20260620 19:05:20.122841 6754 leader_election.cc:288] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 420b44f531184026b7016ea42a45063e (127.1.154.130:38329), 98b1a849d5b4456c9e2972565f5c2268 (127.1.154.131:33993)
I20260620 19:05:20.125030 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.123275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:49091 (local address 127.1.154.131:33993)
0620 19:05:20.123390 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:20.123395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:20.123556 (+ 161us) server_negotiation.cc:407] Connection header received
0620 19:05:20.123693 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:20.123696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:20.123737 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:20.123839 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:20.124646 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:20.124649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:20.124652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:20.124666 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:20.124688 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:20.124696 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:20.124801 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:20.124865 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 19:05:20.124905 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:05:20.125200 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.123158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:60461 (local address 127.1.154.130:38329)
0620 19:05:20.123307 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:20.123312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:20.123508 (+ 196us) server_negotiation.cc:407] Connection header received
0620 19:05:20.123661 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:20.123664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:20.123707 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:20.123795 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:20.124732 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:20.124736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:20.124739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:20.124755 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:20.124779 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:20.124788 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:20.124905 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:20.124968 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 19:05:20.125063 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:05:20.125244 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.123059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:38329 (local address 127.1.154.129:60461)
0620 19:05:20.123392 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:20.123408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:20.123543 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:20.123812 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:20.123815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:20.123829 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:20.124396 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:05:20.124399 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:20.124414 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:05:20.124485 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:20.124890 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:20.124893 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:20.124897 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:05:20.124977 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 19:05:20.125033 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":103,"threads_started":1}
I20260620 19:05:20.125720 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.123151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:33993 (local address 127.1.154.129:49091)
0620 19:05:20.123464 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:20.123475 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:20.123585 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:20.123854 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:20.123857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:20.123870 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:20.124481 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:05:20.124484 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:20.124503 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:05:20.124531 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:20.124786 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:20.124789 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:20.124792 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:05:20.124873 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 19:05:20.124905 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":65,"threads_started":1}
I20260620 19:05:20.126137 6554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3e24fe69ce4dda92d4e4c4b9da23a3" candidate_uuid: "2ebb7862b568434d90432d01ee651bd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "420b44f531184026b7016ea42a45063e" is_pre_election: true
I20260620 19:05:20.126478 6554 raft_consensus.cc:2454] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ebb7862b568434d90432d01ee651bd4 in term 0.
I20260620 19:05:20.126459 6685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3e24fe69ce4dda92d4e4c4b9da23a3" candidate_uuid: "2ebb7862b568434d90432d01ee651bd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b1a849d5b4456c9e2972565f5c2268" is_pre_election: true
I20260620 19:05:20.126703 6685 raft_consensus.cc:2454] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ebb7862b568434d90432d01ee651bd4 in term 0.
I20260620 19:05:20.127102 6357 leader_election.cc:302] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ebb7862b568434d90432d01ee651bd4, 420b44f531184026b7016ea42a45063e; no voters:
I20260620 19:05:20.127427 6754 raft_consensus.cc:2781] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:20.127528 6754 raft_consensus.cc:482] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:20.127578 6754 raft_consensus.cc:3037] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:20.128614 6754 raft_consensus.cc:504] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "420b44f531184026b7016ea42a45063e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38329 } } peers { permanent_uuid: "98b1a849d5b4456c9e2972565f5c2268" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33993 } } peers { permanent_uuid: "2ebb7862b568434d90432d01ee651bd4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45561 } }
I20260620 19:05:20.128985 6754 leader_election.cc:288] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [CANDIDATE]: Term 1 election: Requested vote from peers 420b44f531184026b7016ea42a45063e (127.1.154.130:38329), 98b1a849d5b4456c9e2972565f5c2268 (127.1.154.131:33993)
I20260620 19:05:20.129380 6554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3e24fe69ce4dda92d4e4c4b9da23a3" candidate_uuid: "2ebb7862b568434d90432d01ee651bd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "420b44f531184026b7016ea42a45063e"
I20260620 19:05:20.129544 6554 raft_consensus.cc:3037] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:20.129586 6685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3e24fe69ce4dda92d4e4c4b9da23a3" candidate_uuid: "2ebb7862b568434d90432d01ee651bd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b1a849d5b4456c9e2972565f5c2268"
I20260620 19:05:20.129727 6685 raft_consensus.cc:3037] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:20.130689 6554 raft_consensus.cc:2454] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ebb7862b568434d90432d01ee651bd4 in term 1.
I20260620 19:05:20.130797 6685 raft_consensus.cc:2454] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ebb7862b568434d90432d01ee651bd4 in term 1.
I20260620 19:05:20.130980 6357 leader_election.cc:302] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ebb7862b568434d90432d01ee651bd4, 420b44f531184026b7016ea42a45063e; no voters:
I20260620 19:05:20.131141 6754 raft_consensus.cc:2781] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:20.131351 6754 raft_consensus.cc:686] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [term 1 LEADER]: Becoming Leader. State: Replica: 2ebb7862b568434d90432d01ee651bd4, State: Running, Role: LEADER
I20260620 19:05:20.131664 6754 consensus_queue.cc:237] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "420b44f531184026b7016ea42a45063e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38329 } } peers { permanent_uuid: "98b1a849d5b4456c9e2972565f5c2268" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33993 } } peers { permanent_uuid: "2ebb7862b568434d90432d01ee651bd4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45561 } }
I20260620 19:05:20.135226 6281 catalog_manager.cc:5495] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ebb7862b568434d90432d01ee651bd4 (127.1.154.129). New cstate: current_term: 1 leader_uuid: "2ebb7862b568434d90432d01ee651bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "420b44f531184026b7016ea42a45063e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98b1a849d5b4456c9e2972565f5c2268" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ebb7862b568434d90432d01ee651bd4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45561 } health_report { overall_health: HEALTHY } } }
I20260620 19:05:20.165010 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.162811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.1.154.129:45561)
0620 19:05:20.162946 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:20.162949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:20.163009 (+ 60us) server_negotiation.cc:407] Connection header received
0620 19:05:20.163197 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:20.163199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:20.163258 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:20.163354 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:20.164241 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:20.164245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:20.164248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:20.164267 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:20.164298 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:20.164466 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:20.164612 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:20.164680 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 19:05:20.164729 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:05:20.171962 6554 raft_consensus.cc:1261] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e [term 1 FOLLOWER]: Refusing update from remote peer 2ebb7862b568434d90432d01ee651bd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:20.171962 6685 raft_consensus.cc:1261] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268 [term 1 FOLLOWER]: Refusing update from remote peer 2ebb7862b568434d90432d01ee651bd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:20.172693 6754 consensus_queue.cc:1035] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98b1a849d5b4456c9e2972565f5c2268" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:20.172904 6759 consensus_queue.cc:1035] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 2ebb7862b568434d90432d01ee651bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "420b44f531184026b7016ea42a45063e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:20.181159 6764 mvcc.cc:205] Tried to move back new op lower bound from 7298999583414697984 to 7298999583261765632. Current Snapshot: MvccSnapshot[applied={T|T < 7298999583414697984}]
I20260620 19:05:20.183557 6763 mvcc.cc:205] Tried to move back new op lower bound from 7298999583414697984 to 7298999583261765632. Current Snapshot: MvccSnapshot[applied={T|T < 7298999583414697984 or (T in {7298999583414697984})}]
I20260620 19:05:20.188092 6281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:20.188288 6281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:20.191025 6281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d1ab3fd55e44d1ead90561b37b409c5: Sending DeleteTablet for 3 replicas of tablet 9f3e24fe69ce4dda92d4e4c4b9da23a3
I20260620 19:05:20.192116 6534 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3e24fe69ce4dda92d4e4c4b9da23a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:20 UTC) from {username='slave'} at 127.0.0.1:54428
I20260620 19:05:20.192667 6662 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3e24fe69ce4dda92d4e4c4b9da23a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:20 UTC) from {username='slave'} at 127.0.0.1:59364
I20260620 19:05:20.192842 6403 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3e24fe69ce4dda92d4e4c4b9da23a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:20 UTC) from {username='slave'} at 127.0.0.1:47628
I20260620 19:05:20.193166 6772 tablet_replica.cc:330] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268: stopping tablet replica
I20260620 19:05:20.193245 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 6341
I20260620 19:05:20.194427 6772 raft_consensus.cc:2229] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:20.194773 6772 raft_consensus.cc:2258] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:20.195071 6771 tablet_replica.cc:330] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e: stopping tablet replica
I20260620 19:05:20.195353 6771 raft_consensus.cc:2229] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:20.195602 6771 raft_consensus.cc:2258] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:20.196416 6772 ts_tablet_manager.cc:1905] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:20.196779 6771 ts_tablet_manager.cc:1905] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:20.199656 6771 ts_tablet_manager.cc:1918] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:20.199744 6771 log.cc:1196] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/9f3e24fe69ce4dda92d4e4c4b9da23a3
I20260620 19:05:20.200069 6771 ts_tablet_manager.cc:1939] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 420b44f531184026b7016ea42a45063e: Deleting consensus metadata
I20260620 19:05:20.200380 6772 ts_tablet_manager.cc:1918] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:20.200474 6772 log.cc:1196] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/9f3e24fe69ce4dda92d4e4c4b9da23a3
I20260620 19:05:20.200820 6772 ts_tablet_manager.cc:1939] T 9f3e24fe69ce4dda92d4e4c4b9da23a3 P 98b1a849d5b4456c9e2972565f5c2268: Deleting consensus metadata
I20260620 19:05:20.201370 6266 catalog_manager.cc:4841] TS 420b44f531184026b7016ea42a45063e (127.1.154.130:38329): tablet 9f3e24fe69ce4dda92d4e4c4b9da23a3 (table test-table [id=29571b0da5c74214a010524fd7633d52]) successfully deleted
W20260620 19:05:20.201756 6265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:45561 (error 108)
I20260620 19:05:20.202224 6268 catalog_manager.cc:4841] TS 98b1a849d5b4456c9e2972565f5c2268 (127.1.154.131:33993): tablet 9f3e24fe69ce4dda92d4e4c4b9da23a3 (table test-table [id=29571b0da5c74214a010524fd7633d52]) successfully deleted
I20260620 19:05:20.202622 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 6472
I20260620 19:05:20.203935 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.202680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:45561 (local address 127.0.0.1:47640)
0620 19:05:20.203624 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:20.203776 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":725}
W20260620 19:05:20.206851 6265 catalog_manager.cc:4584] TS 2ebb7862b568434d90432d01ee651bd4 (127.1.154.129:45561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f3e24fe69ce4dda92d4e4c4b9da23a3: Network error: Client connection negotiation failed: client connection to 127.1.154.129:45561: connect: Connection refused (error 111)
I20260620 19:05:20.210322 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 6603
I20260620 19:05:20.215994 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 6250
2026-06-20T19:05:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1073 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 19:05:20.237749 1642 mini_oidc.cc:152] Starting JWKS server
I20260620 19:05:20.239248 1642 webserver.cc:460] Webserver started at https://127.0.0.1:36901/ using document root <none> and password file <none>
I20260620 19:05:20.239399 1642 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:05:20.239614 1642 webserver.cc:460] Webserver started at http://127.0.0.1:45743/ using document root <none> and password file <none>
2026-06-20T19:05:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:20Z Disabled control of system clock
I20260620 19:05:20.248978 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:36069
--webserver_interface=127.1.154.190
--webserver_port=0
--builtin_ntp_servers=127.1.154.148:36457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:36069
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36901/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/testchainca.cert with env {}
W20260620 19:05:20.370519 6780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:20.370808 6780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:20.370851 6780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:20.375149 6780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:20.375217 6780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:20.375233 6780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:20.375250 6780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:20.375268 6780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:05:20.375286 6780 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:05:20.375325 6780 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36901/jwks.json
I20260620 19:05:20.380985 6780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:36457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:36069
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:36069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36901/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:20.382169 6780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:20.383267 6780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:20.392207 6786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:20.392222 6785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:20.392277 6788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:20.393496 6780 server_base.cc:1034] running on GCE node
I20260620 19:05:20.394129 6780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:20.395258 6780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:20.396476 6780 hybrid_clock.cc:648] HybridClock initialized: now 1781982320396457 us; error 36 us; skew 500 ppm
I20260620 19:05:20.398483 6780 webserver.cc:460] Webserver started at http://127.1.154.190:44637/ using document root <none> and password file <none>
I20260620 19:05:20.399076 6780 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:20.399142 6780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:20.399372 6780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:20.401386 6780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "4dc8299462be4b5a952df8abc3521d2e"
format_stamp: "Formatted at 2026-06-20 19:05:20 on dist-test-slave-6wrf"
server_key: "a5e0ee911decbc9aff37fc87703b0767"
server_key_iv: "c00444529c7723795d2538ad093a9575"
server_key_version: "encryptionkey@0"
I20260620 19:05:20.402014 6780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "4dc8299462be4b5a952df8abc3521d2e"
format_stamp: "Formatted at 2026-06-20 19:05:20 on dist-test-slave-6wrf"
server_key: "a5e0ee911decbc9aff37fc87703b0767"
server_key_iv: "c00444529c7723795d2538ad093a9575"
server_key_version: "encryptionkey@0"
I20260620 19:05:20.405851 6780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:05:20.408847 6794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:20.411012 6780 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:05:20.411144 6780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "4dc8299462be4b5a952df8abc3521d2e"
format_stamp: "Formatted at 2026-06-20 19:05:20 on dist-test-slave-6wrf"
server_key: "a5e0ee911decbc9aff37fc87703b0767"
server_key_iv: "c00444529c7723795d2538ad093a9575"
server_key_version: "encryptionkey@0"
I20260620 19:05:20.411340 6780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:20.448489 6780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:20.449293 6780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:20.449496 6780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:20.459074 6780 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:36069
I20260620 19:05:20.459424 6846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:36069 every 8 connection(s)
I20260620 19:05:20.460798 6780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:20.465615 6847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:20.466816 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 6780
I20260620 19:05:20.466965 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:20.467283 1642 external_mini_cluster.cc:1428] Setting key 8fcac4bb37c696b0d51dd6ad5a112d4d
I20260620 19:05:20.473304 6847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e: Bootstrap starting.
I20260620 19:05:20.476552 6847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:20.477552 6847 log.cc:824] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:20.477581 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.468852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.1.154.190:36069)
0620 19:05:20.469555 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:20.469571 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:20.469669 (+ 98us) server_negotiation.cc:407] Connection header received
0620 19:05:20.470336 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:20.470354 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:20.470759 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:20.471116 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:20.471579 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:20.472589 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:20.473579 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:20.474133 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:20.474322 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:20.475292 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:20.475298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:20.475311 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:20.475772 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:20.475835 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:20.475858 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:20.475983 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:20.476379 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 19:05:20.476807 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":130,"threads_started":1}
I20260620 19:05:20.479781 6847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e: No bootstrap required, opened a new log
I20260620 19:05:20.481787 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:05:20.484964 6847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc8299462be4b5a952df8abc3521d2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36069 } }
I20260620 19:05:20.485337 6847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:20.485412 6847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc8299462be4b5a952df8abc3521d2e, State: Initialized, Role: FOLLOWER
I20260620 19:05:20.486019 6847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc8299462be4b5a952df8abc3521d2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36069 } }
I20260620 19:05:20.486219 6847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:20.486300 6847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:20.486435 6847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:20.487619 6847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc8299462be4b5a952df8abc3521d2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36069 } }
I20260620 19:05:20.488072 6847 leader_election.cc:302] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dc8299462be4b5a952df8abc3521d2e; no voters:
I20260620 19:05:20.488498 6847 leader_election.cc:288] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:20.488714 6852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:20.489176 6852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [term 1 LEADER]: Becoming Leader. State: Replica: 4dc8299462be4b5a952df8abc3521d2e, State: Running, Role: LEADER
I20260620 19:05:20.489773 6852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc8299462be4b5a952df8abc3521d2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36069 } }
I20260620 19:05:20.490276 6847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:20.492043 6853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dc8299462be4b5a952df8abc3521d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc8299462be4b5a952df8abc3521d2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36069 } } }
I20260620 19:05:20.492180 6854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dc8299462be4b5a952df8abc3521d2e. Latest consensus state: current_term: 1 leader_uuid: "4dc8299462be4b5a952df8abc3521d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc8299462be4b5a952df8abc3521d2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36069 } } }
I20260620 19:05:20.492390 6853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:20.492412 6854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:20.493212 6863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:20.496754 6863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:20.503316 6863 catalog_manager.cc:1347] Generated new cluster ID: ce93cc494a2b4157ba305e71287417c1
I20260620 19:05:20.503430 6863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:20.514022 6863 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:20.515921 6863 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:20.525503 6863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dc8299462be4b5a952df8abc3521d2e: Generated new TSK 0
I20260620 19:05:20.526484 6863 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 19:05:20.575163 6877 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 19:05:20.575304 6877 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 19:05:20.575831 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.550253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.1.154.190:36069)
0620 19:05:20.550413 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:20.550418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:20.550641 (+ 223us) server_negotiation.cc:407] Connection header received
0620 19:05:20.550899 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:20.550902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:20.550961 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:20.551055 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:05:20.551320 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:20.551896 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:20.553145 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:20.553373 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:20.553460 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:20.554030 (+ 570us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:05:20.575342 (+ 21312us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 19:05:20.575707 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 19:05:20.575943 6850 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 19:05:20.577250 6876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:20.550170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:36069 (local address 127.0.0.1:53996)
0620 19:05:20.550536 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:20.550553 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:20.550694 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:20.551093 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:20.551097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:20.551105 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:05:20.551231 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:20.551238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:20.552004 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:20.552008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:20.552971 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:20.552981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:20.553625 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:20.553630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:20.553726 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:20.553782 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:05:20.575667 (+ 21885us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 19:05:20.575917 (+ 250us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.154.190:36069: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":125,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.154.190:36069: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 19:05:20.578323 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 6780
2026-06-20T19:05:20Z chronyd exiting
I20260620 19:05:20.616950 1642 test_util.cc:175] -----------------------------------------------
I20260620 19:05:20.617085 1642 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982293404112-1642-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (380 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 19:05:20.618479 1642 mini_oidc.cc:152] Starting JWKS server
I20260620 19:05:20.619480 1642 webserver.cc:460] Webserver started at https://127.0.0.1:45079/ using document root <none> and password file <none>
I20260620 19:05:20.619573 1642 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:05:20.619807 1642 webserver.cc:460] Webserver started at http://127.0.0.1:41869/ using document root <none> and password file <none>
2026-06-20T19:05:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:20Z Disabled control of system clock
I20260620 19:05:20.630056 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:46319
--webserver_interface=127.1.154.190
--webserver_port=0
--builtin_ntp_servers=127.1.154.148:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:46319
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45079/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/testchainca.cert with env {}
W20260620 19:05:20.747910 6883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:20.748189 6883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:20.748234 6883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:20.752193 6883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:20.752261 6883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:20.752280 6883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:20.752300 6883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:20.752319 6883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:05:20.752337 6883 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:05:20.752354 6883 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45079/jwks.json
I20260620 19:05:20.757798 6883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:46319
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:46319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45079/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:20.758812 6883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:20.759824 6883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:20.769071 6891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:20.769141 6888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:20.769176 6889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:20.769874 6883 server_base.cc:1034] running on GCE node
I20260620 19:05:20.770414 6883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:20.771562 6883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:20.772871 6883 hybrid_clock.cc:648] HybridClock initialized: now 1781982320772834 us; error 34 us; skew 500 ppm
I20260620 19:05:20.775489 6883 webserver.cc:460] Webserver started at http://127.1.154.190:40035/ using document root <none> and password file <none>
I20260620 19:05:20.776144 6883 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:20.776252 6883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:20.776574 6883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:20.779057 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "c711eb21ab874d719e12aec41d40ff79"
format_stamp: "Formatted at 2026-06-20 19:05:20 on dist-test-slave-6wrf"
server_key: "f4dbc946906b849070b833ed75d44c82"
server_key_iv: "dbdf7e6d6b33ef2f9debdeb96bb58e86"
server_key_version: "encryptionkey@0"
I20260620 19:05:20.779722 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "c711eb21ab874d719e12aec41d40ff79"
format_stamp: "Formatted at 2026-06-20 19:05:20 on dist-test-slave-6wrf"
server_key: "f4dbc946906b849070b833ed75d44c82"
server_key_iv: "dbdf7e6d6b33ef2f9debdeb96bb58e86"
server_key_version: "encryptionkey@0"
I20260620 19:05:20.784296 6883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:05:20.788200 6897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:20.790336 6883 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:05:20.790573 6883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "c711eb21ab874d719e12aec41d40ff79"
format_stamp: "Formatted at 2026-06-20 19:05:20 on dist-test-slave-6wrf"
server_key: "f4dbc946906b849070b833ed75d44c82"
server_key_iv: "dbdf7e6d6b33ef2f9debdeb96bb58e86"
server_key_version: "encryptionkey@0"
I20260620 19:05:20.790731 6883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:20.803360 6883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:20.804111 6883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:20.804327 6883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:20.812058 6883 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:46319
I20260620 19:05:20.812086 6949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:46319 every 8 connection(s)
I20260620 19:05:20.813035 6883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:20.816170 6950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:20.817461 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 6883
I20260620 19:05:20.817572 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:20.817946 1642 external_mini_cluster.cc:1428] Setting key def1e36cba41aeba5a9219c75ffe66a8
I20260620 19:05:20.823035 6950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79: Bootstrap starting.
I20260620 19:05:20.825731 6950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:20.826485 6950 log.cc:824] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:20.828676 6950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79: No bootstrap required, opened a new log
I20260620 19:05:20.828887 6953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.819408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.1.154.190:46319)
0620 19:05:20.820114 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:20.820134 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:20.820176 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:05:20.821070 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:20.821092 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:20.821566 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:20.821936 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:20.822377 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:20.823798 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:20.824760 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:20.825560 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:20.825773 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:20.826668 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:20.826674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:20.826690 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:20.827147 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:20.827198 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:20.827251 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:20.827357 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:20.827699 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 19:05:20.827960 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":107,"threads_started":1}
I20260620 19:05:20.831790 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:05:20.832731 6950 raft_consensus.cc:348] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c711eb21ab874d719e12aec41d40ff79" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46319 } }
I20260620 19:05:20.832942 6950 raft_consensus.cc:374] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:20.832975 6950 raft_consensus.cc:729] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c711eb21ab874d719e12aec41d40ff79, State: Initialized, Role: FOLLOWER
I20260620 19:05:20.833455 6950 consensus_queue.cc:260] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c711eb21ab874d719e12aec41d40ff79" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46319 } }
I20260620 19:05:20.833655 6950 raft_consensus.cc:388] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:20.833721 6950 raft_consensus.cc:482] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:20.833835 6950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:20.834830 6950 raft_consensus.cc:504] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c711eb21ab874d719e12aec41d40ff79" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46319 } }
I20260620 19:05:20.835191 6950 leader_election.cc:302] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c711eb21ab874d719e12aec41d40ff79; no voters:
I20260620 19:05:20.835518 6950 leader_election.cc:288] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:20.835779 6955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:20.836124 6955 raft_consensus.cc:686] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [term 1 LEADER]: Becoming Leader. State: Replica: c711eb21ab874d719e12aec41d40ff79, State: Running, Role: LEADER
I20260620 19:05:20.836745 6950 sys_catalog.cc:564] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:20.836707 6955 consensus_queue.cc:237] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c711eb21ab874d719e12aec41d40ff79" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46319 } }
I20260620 19:05:20.838649 6956 sys_catalog.cc:455] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c711eb21ab874d719e12aec41d40ff79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c711eb21ab874d719e12aec41d40ff79" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46319 } } }
I20260620 19:05:20.838798 6956 sys_catalog.cc:458] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:20.839322 6957 sys_catalog.cc:455] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c711eb21ab874d719e12aec41d40ff79. Latest consensus state: current_term: 1 leader_uuid: "c711eb21ab874d719e12aec41d40ff79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c711eb21ab874d719e12aec41d40ff79" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46319 } } }
I20260620 19:05:20.839872 6957 sys_catalog.cc:458] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79 [sys.catalog]: This master's current role is: LEADER
W20260620 19:05:20.842124 6969 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:05:20.842206 6969 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:05:20.842326 6970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:20.843364 6970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:20.848896 6970 catalog_manager.cc:1347] Generated new cluster ID: f97415604e4e4f359805487fe9e40a9a
I20260620 19:05:20.848954 6970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:20.856279 6970 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:20.857498 6970 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:20.862430 6970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c711eb21ab874d719e12aec41d40ff79: Generated new TSK 0
I20260620 19:05:20.863140 6970 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 19:05:20.890158 6980 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 19:05:20.890273 6980 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 19:05:20.890977 6953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:20.872756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.1.154.190:46319)
0620 19:05:20.872932 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:20.872937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:20.873262 (+ 325us) server_negotiation.cc:407] Connection header received
0620 19:05:20.873458 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:20.873461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:20.873517 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:20.873693 (+ 176us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:05:20.873976 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:20.874800 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:20.875715 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:20.876074 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:20.876181 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:20.876793 (+ 612us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:05:20.890541 (+ 13748us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45079/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 19:05:20.890830 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41730: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45079/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 19:05:20.891042 6953 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41730: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45079/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 19:05:20.891139 6979 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:20.872681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:46319 (local address 127.0.0.1:41730)
0620 19:05:20.873151 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:20.873165 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:20.873312 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:20.873740 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:20.873743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:20.873752 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:05:20.873856 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:20.873860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:20.874941 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:20.874943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:20.875601 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:20.875609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:20.876425 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:20.876431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:20.876536 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:20.876572 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:05:20.890781 (+ 14209us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45079/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 19:05:20.890941 (+ 160us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.154.190:46319: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45079/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":226,"threads_started":1}
I20260620 19:05:20.892051 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 6883
2026-06-20T19:05:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (313 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 19:05:20.931711 1642 mini_oidc.cc:152] Starting JWKS server
I20260620 19:05:20.932591 1642 webserver.cc:460] Webserver started at https://127.0.0.1:36091/ using document root <none> and password file <none>
I20260620 19:05:20.932723 1642 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:05:20.932986 1642 webserver.cc:460] Webserver started at http://127.0.0.1:41517/ using document root <none> and password file <none>
2026-06-20T19:05:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:20Z Disabled control of system clock
I20260620 19:05:20.942929 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:34679
--webserver_interface=127.1.154.190
--webserver_port=0
--builtin_ntp_servers=127.1.154.148:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:34679
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36091/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 19:05:21.050813 6986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:21.051077 6986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:21.051115 6986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:21.055115 6986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:21.055171 6986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:21.055186 6986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:21.055203 6986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:21.055218 6986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:05:21.055276 6986 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:05:21.055305 6986 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36091/jwks.json
I20260620 19:05:21.060029 6986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:34679
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:34679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36091/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:21.061162 6986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:21.062529 6986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:21.071288 6994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:21.071422 6992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:21.071369 6986 server_base.cc:1034] running on GCE node
W20260620 19:05:21.071292 6991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:21.072516 6986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:21.074096 6986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:21.075368 6986 hybrid_clock.cc:648] HybridClock initialized: now 1781982321075341 us; error 36 us; skew 500 ppm
I20260620 19:05:21.077728 6986 webserver.cc:460] Webserver started at http://127.1.154.190:41283/ using document root <none> and password file <none>
I20260620 19:05:21.078323 6986 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:21.078431 6986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:21.078666 6986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:21.080729 6986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "e1b109173c2b41dca98a0b3bec50ba58"
format_stamp: "Formatted at 2026-06-20 19:05:21 on dist-test-slave-6wrf"
server_key: "fa61ad535d5649781fd62fa56d01bba3"
server_key_iv: "49af436b997f0129ed5368d1bd3d8368"
server_key_version: "encryptionkey@0"
I20260620 19:05:21.081321 6986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "e1b109173c2b41dca98a0b3bec50ba58"
format_stamp: "Formatted at 2026-06-20 19:05:21 on dist-test-slave-6wrf"
server_key: "fa61ad535d5649781fd62fa56d01bba3"
server_key_iv: "49af436b997f0129ed5368d1bd3d8368"
server_key_version: "encryptionkey@0"
I20260620 19:05:21.085348 6986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:05:21.088222 7000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:21.089519 6986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:05:21.089728 6986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "e1b109173c2b41dca98a0b3bec50ba58"
format_stamp: "Formatted at 2026-06-20 19:05:21 on dist-test-slave-6wrf"
server_key: "fa61ad535d5649781fd62fa56d01bba3"
server_key_iv: "49af436b997f0129ed5368d1bd3d8368"
server_key_version: "encryptionkey@0"
I20260620 19:05:21.089851 6986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:21.115993 6986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:21.116726 6986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:21.116933 6986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:21.124539 6986 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:34679
I20260620 19:05:21.124559 7052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:34679 every 8 connection(s)
I20260620 19:05:21.125521 6986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:21.128620 7053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:21.129246 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 6986
I20260620 19:05:21.129424 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:21.129927 1642 external_mini_cluster.cc:1428] Setting key d04b8779777c635235fc058f472b9189
I20260620 19:05:21.135387 7053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58: Bootstrap starting.
I20260620 19:05:21.137727 7053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:21.138459 7053 log.cc:824] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:21.140918 7053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58: No bootstrap required, opened a new log
I20260620 19:05:21.142251 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:21.131798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38902 (local address 127.1.154.190:34679)
0620 19:05:21.132652 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:21.132677 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:21.132732 (+ 55us) server_negotiation.cc:407] Connection header received
0620 19:05:21.133704 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:21.133727 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:21.134226 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:21.134739 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:21.135266 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:21.136410 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:21.137370 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:21.138032 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:21.138207 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:21.139495 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:21.139502 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:21.139519 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:21.140000 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:21.140066 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:21.140087 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:21.140177 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:21.140539 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 19:05:21.140944 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":161,"threads_started":1}
I20260620 19:05:21.145360 7053 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b109173c2b41dca98a0b3bec50ba58" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 34679 } }
I20260620 19:05:21.145678 7053 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:21.145731 7053 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b109173c2b41dca98a0b3bec50ba58, State: Initialized, Role: FOLLOWER
I20260620 19:05:21.146006 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:05:21.146546 7053 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b109173c2b41dca98a0b3bec50ba58" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 34679 } }
I20260620 19:05:21.146693 7053 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:21.146777 7053 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:21.146881 7053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:21.148000 7053 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b109173c2b41dca98a0b3bec50ba58" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 34679 } }
I20260620 19:05:21.148327 7053 leader_election.cc:302] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1b109173c2b41dca98a0b3bec50ba58; no voters:
I20260620 19:05:21.148666 7053 leader_election.cc:288] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:21.148888 7058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:21.149211 7058 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [term 1 LEADER]: Becoming Leader. State: Replica: e1b109173c2b41dca98a0b3bec50ba58, State: Running, Role: LEADER
I20260620 19:05:21.149515 7058 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b109173c2b41dca98a0b3bec50ba58" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 34679 } }
I20260620 19:05:21.150342 7053 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:21.150993 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1b109173c2b41dca98a0b3bec50ba58. Latest consensus state: current_term: 1 leader_uuid: "e1b109173c2b41dca98a0b3bec50ba58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b109173c2b41dca98a0b3bec50ba58" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 34679 } } }
I20260620 19:05:21.151053 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1b109173c2b41dca98a0b3bec50ba58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b109173c2b41dca98a0b3bec50ba58" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 34679 } } }
I20260620 19:05:21.151154 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:21.151183 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:21.151530 7064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:21.155126 7064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:21.162081 7064 catalog_manager.cc:1347] Generated new cluster ID: 9912ad9b204d49d1922356ae1dca8947
I20260620 19:05:21.162173 7064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:21.182453 7064 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:21.184098 7064 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:21.193753 7064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1b109173c2b41dca98a0b3bec50ba58: Generated new TSK 0
I20260620 19:05:21.194609 7064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:05:21.218364 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:21.213002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.1.154.190:34679)
0620 19:05:21.213132 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:21.213137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:21.213285 (+ 148us) server_negotiation.cc:407] Connection header received
0620 19:05:21.213428 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:21.213431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:21.213474 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:21.213581 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:05:21.213843 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:21.214373 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:21.215730 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:21.216049 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:21.216212 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:21.216587 (+ 375us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:05:21.218006 (+ 1419us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36091/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 19:05:21.218230 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36091/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 19:05:21.218433 7056 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36091/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 19:05:21.220602 7082 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:21.212879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:34679 (local address 127.0.0.1:38906)
0620 19:05:21.213189 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:21.213204 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:21.213322 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:21.213590 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:21.213609 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:21.213618 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:05:21.213740 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:21.213747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:21.214498 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:21.214502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:21.215513 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:21.215523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:21.216241 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:21.216244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:21.216338 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:21.216366 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:05:21.218178 (+ 1812us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36091/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 19:05:21.220351 (+ 2173us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.154.190:34679: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36091/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":1860,"thread_start_us":101,"threads_started":1}
I20260620 19:05:21.221741 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 6986
E20260620 19:05:21.229403 7083 webserver.cc:533] Webserver: sslize failed:
E20260620 19:05:21.229657 7083 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T19:05:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (329 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T19:05:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:21Z Disabled control of system clock
I20260620 19:05:21.269518 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:38835
--webserver_interface=127.1.154.190
--webserver_port=0
--builtin_ntp_servers=127.1.154.148:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:38835
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/testchainca.cert with env {}
W20260620 19:05:21.382974 7087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:21.383324 7087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:21.383409 7087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:21.387428 7087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:21.387537 7087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:21.387570 7087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:21.387612 7087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:21.387650 7087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:05:21.387714 7087 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:05:21.387753 7087 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 19:05:21.393151 7087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:38835
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:38835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:21.394333 7087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:21.395358 7087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:05:21.404764 7087 server_base.cc:1034] running on GCE node
W20260620 19:05:21.404825 7092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:21.404825 7095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:21.404953 7093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:21.405751 7087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:21.406883 7087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:21.408077 7087 hybrid_clock.cc:648] HybridClock initialized: now 1781982321408048 us; error 46 us; skew 500 ppm
I20260620 19:05:21.410796 7087 webserver.cc:460] Webserver started at http://127.1.154.190:36689/ using document root <none> and password file <none>
I20260620 19:05:21.412031 7087 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:21.412171 7087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:21.412534 7087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:21.415685 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "f69cdc61189a45e3a28e2be60e423988"
format_stamp: "Formatted at 2026-06-20 19:05:21 on dist-test-slave-6wrf"
server_key: "6ffb9ab2e0f193fc56a5f9eedf4d9537"
server_key_iv: "69d4b94655828a4f22478a40b50a8369"
server_key_version: "encryptionkey@0"
I20260620 19:05:21.416388 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "f69cdc61189a45e3a28e2be60e423988"
format_stamp: "Formatted at 2026-06-20 19:05:21 on dist-test-slave-6wrf"
server_key: "6ffb9ab2e0f193fc56a5f9eedf4d9537"
server_key_iv: "69d4b94655828a4f22478a40b50a8369"
server_key_version: "encryptionkey@0"
I20260620 19:05:21.420400 7087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:05:21.423691 7101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:21.425117 7087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:21.425264 7087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "f69cdc61189a45e3a28e2be60e423988"
format_stamp: "Formatted at 2026-06-20 19:05:21 on dist-test-slave-6wrf"
server_key: "6ffb9ab2e0f193fc56a5f9eedf4d9537"
server_key_iv: "69d4b94655828a4f22478a40b50a8369"
server_key_version: "encryptionkey@0"
I20260620 19:05:21.425364 7087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:21.443787 7087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:21.444675 7087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:21.444880 7087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:21.452950 7087 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:38835
I20260620 19:05:21.452997 7153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:38835 every 8 connection(s)
I20260620 19:05:21.453960 7087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:21.456156 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 7087
I20260620 19:05:21.456266 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:21.456578 1642 external_mini_cluster.cc:1428] Setting key 45d1b098cadbb9d67c8fd3c4f567bf1d
I20260620 19:05:21.460237 7154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:21.466741 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:21.458301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.1.154.190:38835)
0620 19:05:21.458967 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:21.458982 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:21.459019 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:05:21.459709 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:21.459725 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:21.460116 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:21.460581 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:21.460992 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:21.462031 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:21.462895 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:21.463387 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:21.463576 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:21.464769 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:21.464774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:21.464786 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:21.465182 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:21.465239 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:21.465270 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:21.465476 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:21.465635 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 19:05:21.465898 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":131,"threads_started":1}
I20260620 19:05:21.467012 7154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988: Bootstrap starting.
I20260620 19:05:21.470024 7154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:21.470801 7154 log.cc:824] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:21.471133 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:05:21.474037 7154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988: No bootstrap required, opened a new log
I20260620 19:05:21.479337 7154 raft_consensus.cc:348] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69cdc61189a45e3a28e2be60e423988" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38835 } }
I20260620 19:05:21.479729 7154 raft_consensus.cc:374] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:21.479857 7154 raft_consensus.cc:729] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69cdc61189a45e3a28e2be60e423988, State: Initialized, Role: FOLLOWER
I20260620 19:05:21.480530 7154 consensus_queue.cc:260] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69cdc61189a45e3a28e2be60e423988" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38835 } }
I20260620 19:05:21.480839 7154 raft_consensus.cc:388] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:21.480950 7154 raft_consensus.cc:482] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:21.481128 7154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:21.483055 7154 raft_consensus.cc:504] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69cdc61189a45e3a28e2be60e423988" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38835 } }
I20260620 19:05:21.483460 7154 leader_election.cc:302] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f69cdc61189a45e3a28e2be60e423988; no voters:
I20260620 19:05:21.483839 7154 leader_election.cc:288] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:21.483968 7159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:21.484218 7159 raft_consensus.cc:686] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [term 1 LEADER]: Becoming Leader. State: Replica: f69cdc61189a45e3a28e2be60e423988, State: Running, Role: LEADER
I20260620 19:05:21.484570 7159 consensus_queue.cc:237] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69cdc61189a45e3a28e2be60e423988" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38835 } }
I20260620 19:05:21.485252 7154 sys_catalog.cc:564] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:21.487507 7161 sys_catalog.cc:455] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f69cdc61189a45e3a28e2be60e423988. Latest consensus state: current_term: 1 leader_uuid: "f69cdc61189a45e3a28e2be60e423988" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69cdc61189a45e3a28e2be60e423988" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38835 } } }
I20260620 19:05:21.487695 7161 sys_catalog.cc:458] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:21.487918 7160 sys_catalog.cc:455] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f69cdc61189a45e3a28e2be60e423988" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69cdc61189a45e3a28e2be60e423988" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38835 } } }
I20260620 19:05:21.488413 7160 sys_catalog.cc:458] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988 [sys.catalog]: This master's current role is: LEADER
W20260620 19:05:21.490906 7174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:05:21.490993 7174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:05:21.491144 7173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:21.492405 7173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:21.499436 7173 catalog_manager.cc:1347] Generated new cluster ID: a512456501da47c686bcabcad1c656ca
I20260620 19:05:21.499543 7173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:21.509382 7173 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:21.510414 7173 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:21.521680 7173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f69cdc61189a45e3a28e2be60e423988: Generated new TSK 0
I20260620 19:05:21.522449 7173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:05:21.602833 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:21.531833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.1.154.190:38835)
0620 19:05:21.531972 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:21.531978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:21.532201 (+ 223us) server_negotiation.cc:407] Connection header received
0620 19:05:21.532423 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:21.532426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:21.532507 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:21.532659 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:05:21.532968 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:21.533525 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:21.534878 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:21.535166 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:21.535409 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:21.535781 (+ 372us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:05:21.602453 (+ 66672us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 19:05:21.602718 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57996: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 19:05:21.602938 7157 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57996: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 19:05:21.603426 7183 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:21.531761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:38835 (local address 127.0.0.1:57996)
0620 19:05:21.532043 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:21.532058 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:21.532178 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:21.532683 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:21.532687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:21.532695 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:05:21.532817 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:21.532824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:21.533707 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:21.533711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:21.534700 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:21.534710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:21.535433 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:21.535437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:21.535535 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:21.535564 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:05:21.602730 (+ 67166us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 19:05:21.602958 (+ 228us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.154.190:38835: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":107,"threads_started":1}
I20260620 19:05:21.604189 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 7087
2026-06-20T19:05:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (363 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (125 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (133 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:05:21 dist-test-slave-6wrf krb5kdc[7195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:05:21 dist-test-slave-6wrf krb5kdc[7195](info): set up 2 sockets
Jun 20 19:05:21 dist-test-slave-6wrf krb5kdc[7195](info): commencing operation
krb5kdc: starting...
W20260620 19:05:22.930123 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:05:23 dist-test-slave-6wrf krb5kdc[7195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:05:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:05:23.069797 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43545
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:43545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:23.188329 7208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:23.188668 7208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:23.188747 7208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:23.192885 7208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:23.192987 7208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:23.193023 7208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:23.193058 7208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:23.193094 7208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:23.198511 7208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:43545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:23.199628 7208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:23.200635 7208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:23.209122 7214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:23.209122 7213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:23.209249 7208 server_base.cc:1034] running on GCE node
W20260620 19:05:23.209123 7216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:23.210484 7208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:23.211793 7208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:23.213258 7208 hybrid_clock.cc:648] HybridClock initialized: now 1781982323213171 us; error 93 us; skew 500 ppm
Jun 20 19:05:23 dist-test-slave-6wrf krb5kdc[7195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982323, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:23.221894 7208 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:05:23.222859 7208 webserver.cc:460] Webserver started at http://127.1.154.190:41111/ using document root <none> and password file <none>
I20260620 19:05:23.223433 7208 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:23.223491 7208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:23.223664 7208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:23.225797 7208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "4c1f2d9644bc412484e3b64f6c2c1073"
format_stamp: "Formatted at 2026-06-20 19:05:23 on dist-test-slave-6wrf"
server_key: "26be0d2fcecc0691b1f41599af379a45"
server_key_iv: "9806dfc4e204152750c897de036a5a36"
server_key_version: "encryptionkey@0"
I20260620 19:05:23.226329 7208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "4c1f2d9644bc412484e3b64f6c2c1073"
format_stamp: "Formatted at 2026-06-20 19:05:23 on dist-test-slave-6wrf"
server_key: "26be0d2fcecc0691b1f41599af379a45"
server_key_iv: "9806dfc4e204152750c897de036a5a36"
server_key_version: "encryptionkey@0"
I20260620 19:05:23.230876 7208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:05:23.233363 7223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:23.234772 7208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:23.234876 7208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "4c1f2d9644bc412484e3b64f6c2c1073"
format_stamp: "Formatted at 2026-06-20 19:05:23 on dist-test-slave-6wrf"
server_key: "26be0d2fcecc0691b1f41599af379a45"
server_key_iv: "9806dfc4e204152750c897de036a5a36"
server_key_version: "encryptionkey@0"
I20260620 19:05:23.234974 7208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:23.274537 7208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:23.278481 7208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:23.278651 7208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:23.285802 7208 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:43545
I20260620 19:05:23.285823 7275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:43545 every 8 connection(s)
I20260620 19:05:23.286767 7208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:23.288683 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 7208
I20260620 19:05:23.288782 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:23.289080 1642 external_mini_cluster.cc:1428] Setting key 0c942705e4e62cbb9bde3fb3851db06f
I20260620 19:05:23.291314 7276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:23.298353 7276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073: Bootstrap starting.
Jun 20 19:05:23 dist-test-slave-6wrf krb5kdc[7195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:23.301003 7276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:23.301963 7276 log.cc:824] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:23.304459 7276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073: No bootstrap required, opened a new log
I20260620 19:05:23.305534 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:23.290773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.1.154.190:43545)
0620 19:05:23.291271 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:23.291290 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:23.291336 (+ 46us) server_negotiation.cc:407] Connection header received
0620 19:05:23.292179 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:23.292201 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:23.292622 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:23.293022 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:23.293841 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:23.295344 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:23.296308 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:23.296999 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:23.297180 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:23.300138 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:23.300144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:23.300159 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:23.300191 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:23.302577 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:23.303094 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:23.303100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:23.303105 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:23.303241 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:23.303531 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:23.303534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:23.303536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:23.304018 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:23.304206 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:23.304473 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 19:05:23.304860 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":143,"threads_started":1}
I20260620 19:05:23.308605 7276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1f2d9644bc412484e3b64f6c2c1073" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43545 } }
I20260620 19:05:23.308853 7276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:23.308941 7276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1f2d9644bc412484e3b64f6c2c1073, State: Initialized, Role: FOLLOWER
I20260620 19:05:23.309401 7276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1f2d9644bc412484e3b64f6c2c1073" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43545 } }
I20260620 19:05:23.309548 7276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:23.309652 7276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:23.309751 7276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:23.310766 7276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1f2d9644bc412484e3b64f6c2c1073" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43545 } }
I20260620 19:05:23.311139 7276 leader_election.cc:302] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c1f2d9644bc412484e3b64f6c2c1073; no voters:
I20260620 19:05:23.311476 7276 leader_election.cc:288] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:23.311638 7281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:23.311887 7281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [term 1 LEADER]: Becoming Leader. State: Replica: 4c1f2d9644bc412484e3b64f6c2c1073, State: Running, Role: LEADER
I20260620 19:05:23.312225 7281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1f2d9644bc412484e3b64f6c2c1073" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43545 } }
I20260620 19:05:23.312738 7276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:23.313887 7283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c1f2d9644bc412484e3b64f6c2c1073. Latest consensus state: current_term: 1 leader_uuid: "4c1f2d9644bc412484e3b64f6c2c1073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1f2d9644bc412484e3b64f6c2c1073" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43545 } } }
I20260620 19:05:23.314013 7283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:23.314317 7282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c1f2d9644bc412484e3b64f6c2c1073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1f2d9644bc412484e3b64f6c2c1073" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43545 } } }
I20260620 19:05:23.314414 7282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:23.314512 7290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:23.317221 7290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:23.323338 7290 catalog_manager.cc:1347] Generated new cluster ID: d9aaa3c199fa46039775b41f28b68598
I20260620 19:05:23.323436 7290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:23.347841 7290 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:23.348749 7290 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:23.355113 7290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c1f2d9644bc412484e3b64f6c2c1073: Generated new TSK 0
I20260620 19:05:23.355768 7290 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:05:23.393792 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43545
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:23.512439 7302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:23.512712 7302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:23.512758 7302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:23.517887 7302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:23.517990 7302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:23.518074 7302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:05:23.525301 7302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:23.526976 7302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:23.528152 7302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:23.538446 7307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:23.538468 7308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:23.538614 7310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:23.538952 7302 server_base.cc:1034] running on GCE node
I20260620 19:05:23.539738 7302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:23.540503 7302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:23.541764 7302 hybrid_clock.cc:648] HybridClock initialized: now 1781982323541725 us; error 54 us; skew 500 ppm
Jun 20 19:05:23 dist-test-slave-6wrf krb5kdc[7195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982323, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:23.544840 7302 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:05:23.545832 7302 webserver.cc:460] Webserver started at http://127.1.154.129:46081/ using document root <none> and password file <none>
I20260620 19:05:23.546396 7302 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:23.546487 7302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:23.546984 7302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:23.549201 7302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "f4f43642c285431c859214b1fe587c2c"
format_stamp: "Formatted at 2026-06-20 19:05:23 on dist-test-slave-6wrf"
server_key: "c6f43bd8b5fc587476606ec9a6cbd2ed"
server_key_iv: "9dfbb20c8097a57611cb31768eab9f1f"
server_key_version: "encryptionkey@0"
I20260620 19:05:23.549834 7302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "f4f43642c285431c859214b1fe587c2c"
format_stamp: "Formatted at 2026-06-20 19:05:23 on dist-test-slave-6wrf"
server_key: "c6f43bd8b5fc587476606ec9a6cbd2ed"
server_key_iv: "9dfbb20c8097a57611cb31768eab9f1f"
server_key_version: "encryptionkey@0"
I20260620 19:05:23.553395 7302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:23.556160 7317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:23.557508 7302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:23.557693 7302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "f4f43642c285431c859214b1fe587c2c"
format_stamp: "Formatted at 2026-06-20 19:05:23 on dist-test-slave-6wrf"
server_key: "c6f43bd8b5fc587476606ec9a6cbd2ed"
server_key_iv: "9dfbb20c8097a57611cb31768eab9f1f"
server_key_version: "encryptionkey@0"
I20260620 19:05:23.557822 7302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:23.584511 7302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:23.587872 7302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:23.588088 7302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:23.588693 7302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:23.589545 7302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:23.589643 7302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:23.589735 7302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:23.589784 7302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:23.600158 7302 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:35019
I20260620 19:05:23.600178 7430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:35019 every 8 connection(s)
I20260620 19:05:23.601182 7302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:05:23.601954 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 7302
I20260620 19:05:23.602087 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:05:23.602388 1642 external_mini_cluster.cc:1428] Setting key ecde11f29fd6725e5c4a44e38ce1f8c7
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
Jun 20 19:05:23 dist-test-slave-6wrf krb5kdc[7195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982323, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:23.620599 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:23.604908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:53587 (local address 127.1.154.190:43545)
0620 19:05:23.605069 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:23.605075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:23.605092 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:05:23.606219 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:23.606223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:23.606280 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:23.606398 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:23.607590 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:23.608177 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:23.609390 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:23.609732 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:23.609854 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:23.616225 (+ 6371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:23.616230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:23.616232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:23.616264 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:23.618010 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:23.618779 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:23.618795 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:23.618797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:23.618881 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:23.619319 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:23.619321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:23.619323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:23.619510 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:23.619594 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:23.620331 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 19:05:23.620452 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 19:05:23.621866 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:23.603801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43545 (local address 127.1.154.129:53587)
0620 19:05:23.604381 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:23.604432 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:23.605991 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:23.606574 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:23.606581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:23.606959 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:23.607439 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:23.607453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:23.608310 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:23.608314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:23.609254 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:23.609264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:23.609891 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:23.609895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:23.610085 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:23.610932 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:23.610957 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:23.615893 (+ 4936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:23.618163 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:23.618174 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:23.618198 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:23.618648 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:23.618987 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:23.618990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:23.618992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:23.619209 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:23.619626 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:23.619636 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:23.620032 (+ 396us) client_negotiation.cc:769] Sending connection context
0620 19:05:23.620281 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 19:05:23.620665 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":164,"threads_started":1}
I20260620 19:05:23.623592 7431 heartbeater.cc:347] Connected to a master server at 127.1.154.190:43545
I20260620 19:05:23.623939 7431 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:23.624472 7431 heartbeater.cc:511] Master 127.1.154.190:43545 requested a full tablet report, sending...
I20260620 19:05:23.626197 7240 ts_manager.cc:194] Registered new tserver with Master: f4f43642c285431c859214b1fe587c2c (127.1.154.129:35019)
I20260620 19:05:23.627409 7240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:53587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:05:23.633042 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43545
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:23.749015 7436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:23.749305 7436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:23.749384 7436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:23.754300 7436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:23.754415 7436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:23.754513 7436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:23.760192 7436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:23.761430 7436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:23.762943 7436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:23.774109 7444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:23.774128 7442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:23.774130 7441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:23.774292 7436 server_base.cc:1034] running on GCE node
I20260620 19:05:23.775033 7436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:23.775839 7436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:23.777053 7436 hybrid_clock.cc:648] HybridClock initialized: now 1781982323776985 us; error 86 us; skew 500 ppm
Jun 20 19:05:23 dist-test-slave-6wrf krb5kdc[7195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982323, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:23.780663 7436 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:05:23.781862 7436 webserver.cc:460] Webserver started at http://127.1.154.130:36387/ using document root <none> and password file <none>
I20260620 19:05:23.782423 7436 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:23.782480 7436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:23.782709 7436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:23.784762 7436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "b56727981e62419380c46be0c5b03f82"
format_stamp: "Formatted at 2026-06-20 19:05:23 on dist-test-slave-6wrf"
server_key: "fb73f517c9e6d441477cce427a5dfd8b"
server_key_iv: "a281fc87cd6aa3fb5d116bb3d0d99e17"
server_key_version: "encryptionkey@0"
I20260620 19:05:23.785353 7436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "b56727981e62419380c46be0c5b03f82"
format_stamp: "Formatted at 2026-06-20 19:05:23 on dist-test-slave-6wrf"
server_key: "fb73f517c9e6d441477cce427a5dfd8b"
server_key_iv: "a281fc87cd6aa3fb5d116bb3d0d99e17"
server_key_version: "encryptionkey@0"
I20260620 19:05:23.790521 7436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 19:05:23.793184 7451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:23.794571 7436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:23.794709 7436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "b56727981e62419380c46be0c5b03f82"
format_stamp: "Formatted at 2026-06-20 19:05:23 on dist-test-slave-6wrf"
server_key: "fb73f517c9e6d441477cce427a5dfd8b"
server_key_iv: "a281fc87cd6aa3fb5d116bb3d0d99e17"
server_key_version: "encryptionkey@0"
I20260620 19:05:23.794827 7436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:23.818071 7436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:23.821291 7436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:23.821496 7436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:23.822100 7436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:23.822932 7436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:23.822983 7436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:23.823021 7436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:23.823036 7436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:23.832655 7436 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:34561
I20260620 19:05:23.832691 7564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:34561 every 8 connection(s)
I20260620 19:05:23.833541 7436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:23.841041 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 7436
I20260620 19:05:23.841166 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:05:23.841521 1642 external_mini_cluster.cc:1428] Setting key d159df3de3ccfe6b6d56e4685077d7a1
Jun 20 19:05:23 dist-test-slave-6wrf krb5kdc[7195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982323, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:23.850009 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:23.835819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:52551 (local address 127.1.154.190:43545)
0620 19:05:23.836028 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:23.836032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:23.837380 (+ 1348us) server_negotiation.cc:407] Connection header received
0620 19:05:23.838864 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:23.838868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:23.838939 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:23.839059 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:23.840190 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:23.840708 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:23.842042 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:23.842262 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:23.842373 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:23.845239 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:23.845243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:23.845246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:23.845278 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:23.846928 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:23.847658 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:23.847662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:23.847666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:23.848019 (+ 353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:23.848581 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:23.848583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:23.848585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:23.848817 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:23.848918 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:23.849746 (+ 828us) server_negotiation.cc:299] Negotiation successful
0620 19:05:23.849853 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 19:05:23.851879 7567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:23.836579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43545 (local address 127.1.154.130:52551)
0620 19:05:23.837189 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:23.837235 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:23.838645 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:23.839225 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:23.839232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:23.839595 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:23.840053 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:23.840065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:23.840830 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:23.840834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:23.841923 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:23.841933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:23.842428 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:23.842432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:23.842613 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:23.843306 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:23.843328 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:23.845033 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:23.847058 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:23.847068 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:23.847087 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:23.847517 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:23.848198 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:23.848204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:23.848206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:23.848426 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:23.848983 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:23.848995 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:23.849382 (+ 387us) client_negotiation.cc:769] Sending connection context
0620 19:05:23.849686 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 19:05:23.850124 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":172,"threads_started":1}
I20260620 19:05:23.853691 7565 heartbeater.cc:347] Connected to a master server at 127.1.154.190:43545
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
I20260620 19:05:23.853967 7565 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:23.854525 7565 heartbeater.cc:511] Master 127.1.154.190:43545 requested a full tablet report, sending...
I20260620 19:05:23.855664 7240 ts_manager.cc:194] Registered new tserver with Master: b56727981e62419380c46be0c5b03f82 (127.1.154.130:34561)
I20260620 19:05:23.856246 7240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:52551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:05:23.874991 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43545
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:23.990479 7570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:23.990752 7570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:23.990839 7570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:23.994961 7570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:23.995025 7570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:23.995085 7570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:24.000499 7570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:24.001648 7570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:24.002702 7570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:24.012627 7576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:24.012627 7575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:24.012864 7578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:24.013100 7570 server_base.cc:1034] running on GCE node
I20260620 19:05:24.013684 7570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:24.014374 7570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:24.015599 7570 hybrid_clock.cc:648] HybridClock initialized: now 1781982324015562 us; error 50 us; skew 500 ppm
Jun 20 19:05:24 dist-test-slave-6wrf krb5kdc[7195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:24.018986 7570 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:05:24.020522 7570 webserver.cc:460] Webserver started at http://127.1.154.131:34895/ using document root <none> and password file <none>
I20260620 19:05:24.021153 7570 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:24.021255 7570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:24.021493 7570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:24.023659 7570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "2db861c4f9dc49cd8c09b568a39a8c48"
format_stamp: "Formatted at 2026-06-20 19:05:24 on dist-test-slave-6wrf"
server_key: "f717fe5a6b37aa9f1d4c2dd113e28e94"
server_key_iv: "c870a786f214fe19aee546363f107598"
server_key_version: "encryptionkey@0"
I20260620 19:05:24.024251 7570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "2db861c4f9dc49cd8c09b568a39a8c48"
format_stamp: "Formatted at 2026-06-20 19:05:24 on dist-test-slave-6wrf"
server_key: "f717fe5a6b37aa9f1d4c2dd113e28e94"
server_key_iv: "c870a786f214fe19aee546363f107598"
server_key_version: "encryptionkey@0"
I20260620 19:05:24.028861 7570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:05:24.031558 7585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:24.032987 7570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 19:05:24.033087 7570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "2db861c4f9dc49cd8c09b568a39a8c48"
format_stamp: "Formatted at 2026-06-20 19:05:24 on dist-test-slave-6wrf"
server_key: "f717fe5a6b37aa9f1d4c2dd113e28e94"
server_key_iv: "c870a786f214fe19aee546363f107598"
server_key_version: "encryptionkey@0"
I20260620 19:05:24.033175 7570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:24.046280 7570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:24.049692 7570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:24.049849 7570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:24.050369 7570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:24.051116 7570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:24.051160 7570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:24.051198 7570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:24.051213 7570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:24.060768 7570 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:40817
I20260620 19:05:24.060741 7698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:40817 every 8 connection(s)
I20260620 19:05:24.061758 7570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:24.062530 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 7570
I20260620 19:05:24.062690 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:05:24.063202 1642 external_mini_cluster.cc:1428] Setting key dd3dd470411d80b5376607fb39c8a4be
Jun 20 19:05:24 dist-test-slave-6wrf krb5kdc[7195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:24.078305 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:24.064260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:35455 (local address 127.1.154.190:43545)
0620 19:05:24.064399 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:24.064405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:24.066005 (+ 1600us) server_negotiation.cc:407] Connection header received
0620 19:05:24.067581 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:24.067584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:24.067629 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:24.067698 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:24.068868 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:24.069494 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:24.070856 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:24.071111 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:24.071203 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:24.074192 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:24.074195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:24.074197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:24.074222 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:24.075688 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:24.076264 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:24.076267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:24.076268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:24.076350 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:24.076755 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:24.076761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:24.076763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:24.077107 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:24.077218 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:24.078004 (+ 786us) server_negotiation.cc:299] Negotiation successful
0620 19:05:24.078144 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:05:24.079648 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:24.064753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43545 (local address 127.1.154.131:35455)
0620 19:05:24.065767 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:24.065855 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:24.067367 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:24.067830 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:24.067836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:24.068200 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:24.068666 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:24.068678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:24.069803 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:24.069806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:24.070707 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:24.070716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:24.071247 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:24.071250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:24.071389 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:24.072026 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:24.072042 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:24.073924 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:24.075819 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:24.075825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:24.075841 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:24.076153 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:24.076455 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:24.076457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:24.076458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:24.076601 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:24.077277 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:24.077288 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:24.077650 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 19:05:24.077919 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 19:05:24.078268 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":151,"threads_started":1}
I20260620 19:05:24.081032 7699 heartbeater.cc:347] Connected to a master server at 127.1.154.190:43545
I20260620 19:05:24.081259 7699 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:24.081766 7699 heartbeater.cc:511] Master 127.1.154.190:43545 requested a full tablet report, sending...
I20260620 19:05:24.082733 7240 ts_manager.cc:194] Registered new tserver with Master: 2db861c4f9dc49cd8c09b568a39a8c48 (127.1.154.131:40817)
I20260620 19:05:24.083637 7240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:35455
I20260620 19:05:24.090787 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/test-admin.keytab.
Jun 20 19:05:24 dist-test-slave-6wrf krb5kdc[7195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:24.106932 1642 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 19:05:24.110491 7706 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:24.108553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42508 (local address 127.1.154.129:35019)
0620 19:05:24.109006 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:24.109022 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:24.109053 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:05:24.109142 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:24.109151 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:24.109333 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:24.109493 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:24.110221 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42508: BlockingRecv error: recv got EOF from 127.0.0.1:42508 (error 108)
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":217,"threads_started":1}
W20260620 19:05:24.110743 1642 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:05:24 dist-test-slave-6wrf krb5kdc[7195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:24.111917 1642 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:05:24 dist-test-slave-6wrf krb5kdc[7195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:24.113211 1642 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:05:24 dist-test-slave-6wrf krb5kdc[7195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.129@KRBTEST.COM
I20260620 19:05:24.123414 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:24.114339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.1.154.129:35019)
0620 19:05:24.114456 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:24.114460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:24.114623 (+ 163us) server_negotiation.cc:407] Connection header received
0620 19:05:24.114739 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:24.114743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:24.114790 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:24.114871 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:24.115434 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:24.116214 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:24.117051 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:24.117432 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:24.117519 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:24.119970 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:24.119979 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:24.119992 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:24.120015 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:24.121475 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:24.121971 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:24.121977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:24.121978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:24.122057 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:24.122334 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:24.122337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:24.122338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:24.122586 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:24.122867 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:24.123041 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 19:05:24.123194 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 19:05:24.128226 7706 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:24.127100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42534 (local address 127.1.154.129:35019)
0620 19:05:24.127264 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:24.127268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:24.127286 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:05:24.127364 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:24.127367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:24.127427 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:24.127534 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:24.128087 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42534: BlockingRecv error: recv got EOF from 127.0.0.1:42534 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 19:05:24.128530 1642 init.cc:244] Error while opening credential cache '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/krb5cc)
Jun 20 19:05:24 dist-test-slave-6wrf krb5kdc[7195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:24.129760 1642 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:05:24 dist-test-slave-6wrf krb5kdc[7195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:24.131063 1642 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:05:24 dist-test-slave-6wrf krb5kdc[7195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.129@KRBTEST.COM
I20260620 19:05:24.141194 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:24.132353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.1.154.129:35019)
0620 19:05:24.132475 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:24.132479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:24.132624 (+ 145us) server_negotiation.cc:407] Connection header received
0620 19:05:24.132787 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:24.132789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:24.132829 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:24.132914 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:24.133410 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:24.134001 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:24.134782 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:24.135015 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:24.135124 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:24.138064 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:24.138068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:24.138070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:24.138094 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:24.139555 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:24.139976 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:24.139981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:24.139982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:24.140061 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:24.140345 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:24.140349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:24.140351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:24.140519 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:24.140671 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:24.140913 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 19:05:24.141032 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 19:05:24.144560 1642 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982293404112-1642-0/krb5kdc/krb5cc' of type FILE
W20260620 19:05:24.144603 7706 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:24.143491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.1.154.129:35019)
0620 19:05:24.143587 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:24.143590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:24.143637 (+ 47us) server_negotiation.cc:407] Connection header received
0620 19:05:24.143751 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:24.143752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:24.143786 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:24.143846 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:24.144414 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42548: BlockingRecv error: recv got EOF from 127.0.0.1:42548 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 19:05:24 dist-test-slave-6wrf krb5kdc[7195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:24.145826 1642 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:05:24 dist-test-slave-6wrf krb5kdc[7195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:24.146678 1642 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:05:24 dist-test-slave-6wrf krb5kdc[7195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.129@KRBTEST.COM
I20260620 19:05:24.156106 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:24.147604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.1.154.129:35019)
0620 19:05:24.147742 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:24.147748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:24.147837 (+ 89us) server_negotiation.cc:407] Connection header received
0620 19:05:24.147987 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:24.147991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:24.148031 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:24.148125 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:24.148609 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:24.149107 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:24.149961 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:24.150197 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:24.150312 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:24.152520 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:24.152524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:24.152527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:24.152557 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:24.154491 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:24.154928 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:24.154933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:24.154935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:24.155014 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:24.155288 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:24.155293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:24.155295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:24.155452 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:24.155611 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:24.155831 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 19:05:24.155942 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:05:24.158070 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 7302
I20260620 19:05:24.164273 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 7436
I20260620 19:05:24.170733 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 7570
I20260620 19:05:24.176294 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 7208
2026-06-20T19:05:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2314 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:05:24 dist-test-slave-6wrf krb5kdc[7719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:05:24 dist-test-slave-6wrf krb5kdc[7719](info): set up 2 sockets
Jun 20 19:05:24 dist-test-slave-6wrf krb5kdc[7719](info): commencing operation
krb5kdc: starting...
W20260620 19:05:25.246093 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:05:25 dist-test-slave-6wrf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:05:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.190.keytab.
Entry for principal oryx/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.190.keytab.
I20260620 19:05:25.384892 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:42505
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.190.keytab
--principal=oryx/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:42505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:25.508564 7732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:25.509060 7732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:25.509136 7732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:25.513545 7732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:25.513674 7732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:25.513724 7732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:25.513762 7732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:25.513798 7732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:25.519408 7732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:42505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.190.keytab
--principal=oryx/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:42505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:25.520567 7732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:25.521687 7732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:25.531137 7737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:25.531164 7738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:25.531260 7740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:25.531849 7732 server_base.cc:1034] running on GCE node
I20260620 19:05:25.532631 7732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:25.534281 7732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:25.535490 7732 hybrid_clock.cc:648] HybridClock initialized: now 1781982325535467 us; error 34 us; skew 500 ppm
Jun 20 19:05:25 dist-test-slave-6wrf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982325, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:25.539060 7732 init.cc:377] Logged in from keytab as oryx/127.1.154.190@KRBTEST.COM (short username oryx)
I20260620 19:05:25.540424 7732 webserver.cc:460] Webserver started at http://127.1.154.190:36849/ using document root <none> and password file <none>
I20260620 19:05:25.541235 7732 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:25.541350 7732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:25.541695 7732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:25.544863 7732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "eeb660d158aa4c4bb950a43b440f194e"
format_stamp: "Formatted at 2026-06-20 19:05:25 on dist-test-slave-6wrf"
server_key: "44b12b656336ba6ab1abbbab25e05958"
server_key_iv: "2d102b98a158f24a0913e57eb01cbe76"
server_key_version: "encryptionkey@0"
I20260620 19:05:25.545583 7732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "eeb660d158aa4c4bb950a43b440f194e"
format_stamp: "Formatted at 2026-06-20 19:05:25 on dist-test-slave-6wrf"
server_key: "44b12b656336ba6ab1abbbab25e05958"
server_key_iv: "2d102b98a158f24a0913e57eb01cbe76"
server_key_version: "encryptionkey@0"
I20260620 19:05:25.549924 7732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 19:05:25.552883 7747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:25.554709 7732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:25.554868 7732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "eeb660d158aa4c4bb950a43b440f194e"
format_stamp: "Formatted at 2026-06-20 19:05:25 on dist-test-slave-6wrf"
server_key: "44b12b656336ba6ab1abbbab25e05958"
server_key_iv: "2d102b98a158f24a0913e57eb01cbe76"
server_key_version: "encryptionkey@0"
I20260620 19:05:25.555003 7732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:25.574069 7732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:25.577718 7732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:25.577896 7732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:25.585160 7732 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:42505
I20260620 19:05:25.585472 7799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:42505 every 8 connection(s)
I20260620 19:05:25.586262 7732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:25.589071 7800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:25.593420 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 7732
I20260620 19:05:25.593555 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:25.593894 1642 external_mini_cluster.cc:1428] Setting key 6e9b014f491c90409b8191810fca7372
I20260620 19:05:25.596989 7800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e: Bootstrap starting.
I20260620 19:05:25.599705 7800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:25.600560 7800 log.cc:824] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:25.603015 7800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e: No bootstrap required, opened a new log
Jun 20 19:05:25 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
I20260620 19:05:25.607123 7800 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } }
I20260620 19:05:25.607439 7800 raft_consensus.cc:374] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:25.607496 7800 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb660d158aa4c4bb950a43b440f194e, State: Initialized, Role: FOLLOWER
I20260620 19:05:25.608012 7800 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } }
I20260620 19:05:25.608187 7800 raft_consensus.cc:388] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:25.608261 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:25.608350 7800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:25.609673 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } }
I20260620 19:05:25.610054 7800 leader_election.cc:302] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eeb660d158aa4c4bb950a43b440f194e; no voters:
I20260620 19:05:25.610422 7800 leader_election.cc:288] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:25.610586 7805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:25.610818 7805 raft_consensus.cc:686] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [term 1 LEADER]: Becoming Leader. State: Replica: eeb660d158aa4c4bb950a43b440f194e, State: Running, Role: LEADER
I20260620 19:05:25.611174 7805 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } }
I20260620 19:05:25.611685 7800 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:25.612154 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:25.595598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40876 (local address 127.1.154.190:42505)
0620 19:05:25.596074 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:25.596093 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:25.596135 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:05:25.596997 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:25.597019 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:25.597471 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:25.597866 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:25.598965 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:25.600460 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:25.601771 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:25.602430 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:25.602594 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:25.605329 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:25.605336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:25.605352 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:25.605382 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:25.608306 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:25.608897 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:25.608904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:25.608910 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:25.609058 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:25.609396 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:25.609399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:25.609400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:25.609965 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:25.610117 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:25.610465 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 19:05:25.610891 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":142,"threads_started":1}
I20260620 19:05:25.613840 7806 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } }
I20260620 19:05:25.614197 7806 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:25.614108 7805 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeb660d158aa4c4bb950a43b440f194e. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } }
I20260620 19:05:25.614632 7805 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: This master's current role is: LEADER
W20260620 19:05:25.616768 7819 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:05:25.616865 7819 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:05:25.616961 7820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:25.617761 7820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:25.623816 7820 catalog_manager.cc:1347] Generated new cluster ID: 362b19a36271485bb8eca1f5ab51d9ab
I20260620 19:05:25.623946 7820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:25.658636 7820 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:25.660351 7820 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:25.670277 7820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e: Generated new TSK 0
I20260620 19:05:25.671175 7820 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.129.keytab.
Entry for principal oryx/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.129.keytab.
I20260620 19:05:25.705847 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:42505
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.129.keytab
--principal=oryx/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:25.820752 7826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:25.821055 7826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:25.821132 7826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:25.825218 7826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:25.825279 7826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:25.825366 7826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:05:25.830749 7826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.129.keytab
--principal=oryx/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:25.831794 7826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:25.832764 7826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:25.842217 7834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:25.842502 7826 server_base.cc:1034] running on GCE node
W20260620 19:05:25.842274 7831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:25.842221 7832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:25.843001 7826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:25.843633 7826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:25.844844 7826 hybrid_clock.cc:648] HybridClock initialized: now 1781982325844832 us; error 26 us; skew 500 ppm
Jun 20 19:05:25 dist-test-slave-6wrf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982325, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:25.847744 7826 init.cc:377] Logged in from keytab as oryx/127.1.154.129@KRBTEST.COM (short username oryx)
I20260620 19:05:25.848603 7826 webserver.cc:460] Webserver started at http://127.1.154.129:44331/ using document root <none> and password file <none>
I20260620 19:05:25.849164 7826 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:25.849252 7826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:25.849484 7826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:25.851490 7826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "70b757ebb93849a9836c58f9969cccbc"
format_stamp: "Formatted at 2026-06-20 19:05:25 on dist-test-slave-6wrf"
server_key: "5f5047162a9d75706cd36949157548b3"
server_key_iv: "8e08bcf5c69e93a7185dce340c759f83"
server_key_version: "encryptionkey@0"
I20260620 19:05:25.852030 7826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "70b757ebb93849a9836c58f9969cccbc"
format_stamp: "Formatted at 2026-06-20 19:05:25 on dist-test-slave-6wrf"
server_key: "5f5047162a9d75706cd36949157548b3"
server_key_iv: "8e08bcf5c69e93a7185dce340c759f83"
server_key_version: "encryptionkey@0"
I20260620 19:05:25.856019 7826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:05:25.858590 7841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:25.860245 7826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:25.860389 7826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "70b757ebb93849a9836c58f9969cccbc"
format_stamp: "Formatted at 2026-06-20 19:05:25 on dist-test-slave-6wrf"
server_key: "5f5047162a9d75706cd36949157548b3"
server_key_iv: "8e08bcf5c69e93a7185dce340c759f83"
server_key_version: "encryptionkey@0"
I20260620 19:05:25.860533 7826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:25.873855 7826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:25.877003 7826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:25.877200 7826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:25.877802 7826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:25.878546 7826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:25.878592 7826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:25.878629 7826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:25.878647 7826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:25.890341 7826 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:45115
I20260620 19:05:25.890359 7954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:45115 every 8 connection(s)
I20260620 19:05:25.891219 7826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:05:25.893864 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 7826
I20260620 19:05:25.894212 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:05:25.894637 1642 external_mini_cluster.cc:1428] Setting key 757a6d3c00b75f5a46f943633f5f6299
Jun 20 19:05:25 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982325, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.1.154.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.154.130@KRBTEST.COM" created.
I20260620 19:05:25.908749 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:25.893005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:59357 (local address 127.1.154.190:42505)
0620 19:05:25.893155 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:25.893160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:25.894657 (+ 1497us) server_negotiation.cc:407] Connection header received
0620 19:05:25.896535 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:25.896538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:25.896580 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:25.896651 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:25.898320 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:25.899213 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:25.901487 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:25.901773 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:25.901867 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:25.904780 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:25.904784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:25.904787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:25.904809 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:25.906379 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:25.906972 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:25.906975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:25.906976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:25.907061 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:25.907467 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:25.907470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:25.907471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:25.907694 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:25.907787 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:25.908337 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 19:05:25.908479 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:05:25.909874 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:25.893414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.129:59357)
0620 19:05:25.894314 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:25.894473 (+ 159us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:25.896310 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:25.896840 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:25.896849 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:25.897380 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:25.898102 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:25.898125 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:25.899522 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:25.899531 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:25.901336 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:25.901351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:25.901944 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:25.901950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:25.902188 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:25.902847 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:25.902874 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:25.904604 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:25.906491 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:25.906495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:25.906507 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:25.906841 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:25.907175 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:25.907178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:25.907180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:25.907327 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:25.907812 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:25.907817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:25.908058 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 19:05:25.908224 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 19:05:25.908611 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":179,"threads_started":1}
I20260620 19:05:25.912184 7955 heartbeater.cc:347] Connected to a master server at 127.1.154.190:42505
I20260620 19:05:25.912613 7955 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:25.913328 7955 heartbeater.cc:511] Master 127.1.154.190:42505 requested a full tablet report, sending...
I20260620 19:05:25.915385 7764 ts_manager.cc:194] Registered new tserver with Master: 70b757ebb93849a9836c58f9969cccbc (127.1.154.129:45115)
I20260620 19:05:25.916787 7764 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:59357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.130.keytab.
Entry for principal oryx/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.130.keytab.
I20260620 19:05:25.926115 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:42505
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.130.keytab
--principal=oryx/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:26.051573 7960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:26.051862 7960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:26.051906 7960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:26.056188 7960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:26.056267 7960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:26.056327 7960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:26.061836 7960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.130.keytab
--principal=oryx/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:26.062883 7960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:26.064025 7960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:26.074713 7966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:26.074677 7968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:26.074685 7965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:26.074996 7960 server_base.cc:1034] running on GCE node
I20260620 19:05:26.075469 7960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:26.076122 7960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:26.077314 7960 hybrid_clock.cc:648] HybridClock initialized: now 1781982326077273 us; error 58 us; skew 500 ppm
Jun 20 19:05:26 dist-test-slave-6wrf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982326, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:26.080967 7960 init.cc:377] Logged in from keytab as oryx/127.1.154.130@KRBTEST.COM (short username oryx)
I20260620 19:05:26.082067 7960 webserver.cc:460] Webserver started at http://127.1.154.130:45687/ using document root <none> and password file <none>
I20260620 19:05:26.082647 7960 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:26.082739 7960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:26.082957 7960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:26.085175 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "d9095510e212429c884037c06eb9b368"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "65c678e0c3ed0f06e0746b57fb27413a"
server_key_iv: "dda1b8c300e01bff3f91666ca3dec513"
server_key_version: "encryptionkey@0"
I20260620 19:05:26.085958 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "d9095510e212429c884037c06eb9b368"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "65c678e0c3ed0f06e0746b57fb27413a"
server_key_iv: "dda1b8c300e01bff3f91666ca3dec513"
server_key_version: "encryptionkey@0"
I20260620 19:05:26.089733 7960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:05:26.092629 7975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:26.094164 7960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:26.094300 7960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "d9095510e212429c884037c06eb9b368"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "65c678e0c3ed0f06e0746b57fb27413a"
server_key_iv: "dda1b8c300e01bff3f91666ca3dec513"
server_key_version: "encryptionkey@0"
I20260620 19:05:26.094424 7960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:26.114549 7960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:26.117954 7960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:26.118155 7960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:26.118738 7960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:26.119653 7960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:26.119731 7960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:26.119802 7960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:26.119856 7960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:26.129344 7960 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:41521
I20260620 19:05:26.129364 8088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:41521 every 8 connection(s)
I20260620 19:05:26.130620 7960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:26.135125 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 7960
I20260620 19:05:26.135221 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:05:26.135509 1642 external_mini_cluster.cc:1428] Setting key 4fec52cae9c7252cca5e417dd10d6b10
Jun 20 19:05:26 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982326, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.1.154.131@KRBTEST.COM; defaulting to no policy
I20260620 19:05:26.147541 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.132972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:57463 (local address 127.1.154.190:42505)
0620 19:05:26.133126 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:26.133130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:26.133939 (+ 809us) server_negotiation.cc:407] Connection header received
0620 19:05:26.135390 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:26.135394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:26.135454 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:26.135585 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:26.136939 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.137744 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.139371 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.139594 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.139687 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.143683 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:26.143686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:26.143688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:26.143711 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:26.145412 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.146054 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.146056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.146057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.146133 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.146489 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.146491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.146492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.146691 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:26.146807 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:26.147319 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 19:05:26.147422 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.154.131@KRBTEST.COM" created.
I20260620 19:05:26.148623 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.133255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.130:57463)
0620 19:05:26.133783 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:26.133826 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:26.135115 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:26.135765 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:26.135772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:26.136253 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:26.136781 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.136799 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.137901 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.137904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.139195 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.139207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.139888 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.139895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.140237 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.140917 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:26.140942 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:26.143465 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:26.145547 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.145556 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.145575 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.145961 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.146241 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.146244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.146245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.146394 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.146812 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:26.146817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:26.147084 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 19:05:26.147240 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 19:05:26.147524 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":128,"threads_started":1}
I20260620 19:05:26.149887 8089 heartbeater.cc:347] Connected to a master server at 127.1.154.190:42505
I20260620 19:05:26.150146 8089 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:26.150678 8089 heartbeater.cc:511] Master 127.1.154.190:42505 requested a full tablet report, sending...
I20260620 19:05:26.151773 7764 ts_manager.cc:194] Registered new tserver with Master: d9095510e212429c884037c06eb9b368 (127.1.154.130:41521)
I20260620 19:05:26.152400 7764 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:57463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.131.keytab.
Entry for principal oryx/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.131.keytab.
I20260620 19:05:26.168911 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:42505
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.131.keytab
--principal=oryx/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:26.288959 8094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:26.289283 8094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:26.289368 8094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:26.293423 8094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:26.293529 8094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:26.293638 8094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:26.300073 8094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.131.keytab
--principal=oryx/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:26.301538 8094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:26.302901 8094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:26.315676 8100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:26.315775 8094 server_base.cc:1034] running on GCE node
W20260620 19:05:26.315676 8099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:26.315712 8102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:26.316735 8094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:26.317464 8094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:26.318706 8094 hybrid_clock.cc:648] HybridClock initialized: now 1781982326318686 us; error 63 us; skew 500 ppm
Jun 20 19:05:26 dist-test-slave-6wrf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982326, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:26.323136 8094 init.cc:377] Logged in from keytab as oryx/127.1.154.131@KRBTEST.COM (short username oryx)
I20260620 19:05:26.324182 8094 webserver.cc:460] Webserver started at http://127.1.154.131:32955/ using document root <none> and password file <none>
I20260620 19:05:26.324776 8094 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:26.324846 8094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:26.325060 8094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:26.327392 8094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "d6820357b6174d018dad094c0af49d96"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "097589b7725d1b635a4cdc43ef4c3296"
server_key_iv: "6b8a2628e3d74026fc07b558472b1543"
server_key_version: "encryptionkey@0"
I20260620 19:05:26.328368 8094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "d6820357b6174d018dad094c0af49d96"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "097589b7725d1b635a4cdc43ef4c3296"
server_key_iv: "6b8a2628e3d74026fc07b558472b1543"
server_key_version: "encryptionkey@0"
I20260620 19:05:26.333969 8094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 19:05:26.336764 8109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:26.338086 8094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:26.338240 8094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "d6820357b6174d018dad094c0af49d96"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "097589b7725d1b635a4cdc43ef4c3296"
server_key_iv: "6b8a2628e3d74026fc07b558472b1543"
server_key_version: "encryptionkey@0"
I20260620 19:05:26.338371 8094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:26.353484 8094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:26.357952 8094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:26.358321 8094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:26.359200 8094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:26.360369 8094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:26.360440 8094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:26.360514 8094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:26.360558 8094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:26.372715 8094 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:43351
I20260620 19:05:26.373119 8222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:43351 every 8 connection(s)
I20260620 19:05:26.373991 8094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:26.376899 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 8094
I20260620 19:05:26.377034 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:05:26.377360 1642 external_mini_cluster.cc:1428] Setting key 235fa39d587731497066f669c56618bc
Jun 20 19:05:26 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982326, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
I20260620 19:05:26.389446 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.376208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:44817 (local address 127.1.154.190:42505)
0620 19:05:26.376386 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:26.376391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:26.377736 (+ 1345us) server_negotiation.cc:407] Connection header received
0620 19:05:26.379225 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:26.379228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:26.379280 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:26.379415 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:26.380708 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.381239 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.382529 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.382745 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.382865 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.385542 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:26.385545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:26.385547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:26.385568 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:26.387371 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.387927 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.387933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.387934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.388015 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.388364 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.388367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.388368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.388544 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:26.388652 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:26.389149 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 19:05:26.389273 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 19:05:26.390276 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.376537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.131:44817)
0620 19:05:26.377405 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:26.377527 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:26.378942 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:26.379603 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:26.379609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:26.379989 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:26.380567 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.380581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.381367 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.381372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.382400 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.382410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.382903 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.382907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.383072 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.383650 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:26.383666 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:26.385357 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:26.387518 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.387524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.387540 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.387814 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.388113 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.388117 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.388119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.388256 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.388649 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:26.388656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:26.388910 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 19:05:26.389100 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 19:05:26.389420 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":191,"threads_started":1}
I20260620 19:05:26.391788 8223 heartbeater.cc:347] Connected to a master server at 127.1.154.190:42505
I20260620 19:05:26.392303 8223 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:26.393069 8223 heartbeater.cc:511] Master 127.1.154.190:42505 requested a full tablet report, sending...
I20260620 19:05:26.394450 7764 ts_manager.cc:194] Registered new tserver with Master: d6820357b6174d018dad094c0af49d96 (127.1.154.131:43351)
I20260620 19:05:26.395011 7764 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:44817
I20260620 19:05:26.405261 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.154.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.154.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.154.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.189.keytab.
Entry for principal oryx/127.1.154.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.189.keytab.
I20260620 19:05:26.433753 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.189:45641
--webserver_interface=127.1.154.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.189.keytab
--principal=oryx/127.1.154.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:42505,127.1.154.189:45641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:26.548344 8228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:26.548667 8228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:26.548743 8228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:26.552769 8228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:26.552867 8228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:26.552904 8228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:26.552965 8228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:26.553014 8228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:26.558509 8228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:42505,127.1.154.189:45641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.189.keytab
--principal=oryx/127.1.154.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.189:45641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.154.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
W20260620 19:05:26.558760 8228 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.154.190:42505,127.1.154.189:45641'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 19:05:26.559738 8228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:26.560786 8228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:26.570255 8236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:26.570276 8234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:26.570276 8233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:26.570501 8228 server_base.cc:1034] running on GCE node
I20260620 19:05:26.571230 8228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:26.572389 8228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:26.573661 8228 hybrid_clock.cc:648] HybridClock initialized: now 1781982326573554 us; error 61 us; skew 500 ppm
Jun 20 19:05:26 dist-test-slave-6wrf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982326, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:26.576782 8228 init.cc:377] Logged in from keytab as oryx/127.1.154.189@KRBTEST.COM (short username oryx)
I20260620 19:05:26.577735 8228 webserver.cc:460] Webserver started at http://127.1.154.189:34045/ using document root <none> and password file <none>
I20260620 19:05:26.578321 8228 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:26.578387 8228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:26.578560 8228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:26.580801 8228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/data/instance:
uuid: "1a0454ecded94ee4bc4f43bef6153095"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "b728cbd65ec7fa6002fcf363d97b239d"
server_key_iv: "4f992afcb6ec707a02760f69e3db4c3f"
server_key_version: "encryptionkey@0"
I20260620 19:05:26.581499 8228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/wal/instance:
uuid: "1a0454ecded94ee4bc4f43bef6153095"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "b728cbd65ec7fa6002fcf363d97b239d"
server_key_iv: "4f992afcb6ec707a02760f69e3db4c3f"
server_key_version: "encryptionkey@0"
I20260620 19:05:26.586243 8228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:05:26.589087 8243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:26.591008 8228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:26.591183 8228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/wal
uuid: "1a0454ecded94ee4bc4f43bef6153095"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "b728cbd65ec7fa6002fcf363d97b239d"
server_key_iv: "4f992afcb6ec707a02760f69e3db4c3f"
server_key_version: "encryptionkey@0"
I20260620 19:05:26.591320 8228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:26.611280 8228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:26.615068 8228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:26.615341 8228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:26.623551 8228 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.189:45641
I20260620 19:05:26.623781 8295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.189:45641 every 8 connection(s)
I20260620 19:05:26.624647 8228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/data/info.pb
I20260620 19:05:26.627740 8296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:26.631313 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 8228
I20260620 19:05:26.631862 8296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:05:26 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982326, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.189@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
Jun 20 19:05:26 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.154.189@KRBTEST.COM
I20260620 19:05:26.647487 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.633946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.1.154.190:42505)
0620 19:05:26.634092 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:26.634097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:26.634360 (+ 263us) server_negotiation.cc:407] Connection header received
0620 19:05:26.635012 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:26.635016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:26.635076 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:26.635186 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:26.636174 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.636736 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.637961 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.638255 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.638344 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.642036 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:26.642042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:26.642049 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:26.642090 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:26.644407 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.644982 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.644986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.644989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.645093 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.645731 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.645739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.645742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.646327 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:26.646462 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:26.647070 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 19:05:26.647260 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 19:05:26.648530 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.633837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.0.0.1:40888)
0620 19:05:26.634195 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:26.634233 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:26.634815 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:26.635256 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:26.635261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:26.635727 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:26.636010 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.636029 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.636873 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.636877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.637838 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.637848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.638477 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.638482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.638864 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.639883 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:26.639907 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:26.641770 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:26.644562 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.644570 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.644579 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.644863 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.645200 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.645202 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.645203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.645425 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.646461 (+ 1036us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:26.646467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:26.646758 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 19:05:26.646979 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 19:05:26.647360 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":4,"thread_start_us":117,"threads_started":1}
I20260620 19:05:26.648532 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.633312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52754 (local address 127.1.154.189:45641)
0620 19:05:26.633809 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:26.633830 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:26.633869 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:05:26.634799 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:26.634817 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:26.635005 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:26.635224 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:26.636030 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.637047 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.638148 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.639014 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.639187 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.642906 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:26.642919 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:26.642948 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:26.643129 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:26.645178 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.645807 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.645819 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.645824 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.645944 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.646330 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.646334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.646336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.646782 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:26.647064 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:26.647251 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 19:05:26.647446 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":359,"thread_start_us":132,"threads_started":1}
I20260620 19:05:26.650158 8296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:05:26 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982326, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.189@KRBTEST.COM for oryx/127.1.154.189@KRBTEST.COM
I20260620 19:05:26.659755 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.650901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:45641 (local address 127.0.0.1:52770)
0620 19:05:26.651067 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:26.651091 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:26.651190 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:26.651468 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:26.651471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:26.651631 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:26.651727 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.651732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.652869 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.652872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.653413 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.653420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.654271 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.654274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.654349 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.654906 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:26.654917 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:26.656197 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:26.658062 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.658066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.658068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.658390 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.658772 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.658779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.658782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.658896 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.659312 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:26.659314 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:26.659392 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 19:05:26.659487 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 19:05:26.659619 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 19:05:26.659852 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.650912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.1.154.189:45641)
0620 19:05:26.651053 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:26.651063 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:26.651165 (+ 102us) server_negotiation.cc:407] Connection header received
0620 19:05:26.651270 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:26.651273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:26.651313 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:26.651413 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:26.651858 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.652713 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.653563 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.654144 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.654232 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.656322 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:26.656324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:26.656326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:26.656345 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:26.657876 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.658554 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.658557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.658559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.658643 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.658986 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.658988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.658989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.659163 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:26.659241 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:26.659619 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 19:05:26.659717 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 19:05:26.663928 8296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: Bootstrap starting.
I20260620 19:05:26.667004 8296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:26.667732 8296 log.cc:824] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:26.669852 8296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: No bootstrap required, opened a new log
I20260620 19:05:26.673123 8296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } }
I20260620 19:05:26.673329 8296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:26.673365 8296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a0454ecded94ee4bc4f43bef6153095, State: Initialized, Role: FOLLOWER
I20260620 19:05:26.673830 8296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } }
I20260620 19:05:26.674266 8302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } } }
I20260620 19:05:26.674412 8302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:05:26.675014 8296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 19:05:26.679837 8313 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:05:26.679924 8313 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:05:26.682272 8228 master_runner.cc:428] Detected that this master 1a0454ecded94ee4bc4f43bef6153095 is joining an existing cluster
I20260620 19:05:26.682344 8228 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.154.189:45641
I20260620 19:05:26.693818 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.684711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.1.154.189:45641)
0620 19:05:26.684815 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:26.684819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:26.685241 (+ 422us) server_negotiation.cc:407] Connection header received
0620 19:05:26.685389 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:26.685395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:26.685431 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:26.685502 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:26.686140 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.686891 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.688106 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.688393 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.688597 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.690643 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:26.690647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:26.690650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:26.690678 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:26.692173 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.692602 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.692606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.692608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.692688 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.692965 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.692971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.692974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.693140 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:26.693238 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:26.693482 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 19:05:26.693663 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":42}
I20260620 19:05:26.693828 8320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.684852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:45641 (local address 127.0.0.1:52776)
0620 19:05:26.685149 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:26.685162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:26.685292 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:26.685536 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:26.685539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:26.685903 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:26.686041 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.686048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.687016 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.687022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.687925 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.687935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.688619 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.688622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.688720 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.690021 (+ 1301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:26.690041 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:26.690510 (+ 469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:26.692299 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.692303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.692305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.692511 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.692779 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.692784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.692789 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.692874 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.693237 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:26.693240 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:26.693338 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 19:05:26.693441 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 19:05:26.693582 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":88,"threads_started":1}
I20260620 19:05:26.694125 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.684671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40904 (local address 127.1.154.190:42505)
0620 19:05:26.684814 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:26.684819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:26.685109 (+ 290us) server_negotiation.cc:407] Connection header received
0620 19:05:26.685263 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:26.685268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:26.685315 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:26.685405 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:26.686065 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.686891 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.689131 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.689429 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.689549 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.690823 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:26.690827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:26.690829 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:26.690853 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:26.692435 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.692837 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.692840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.692842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.692918 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.693199 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.693202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.693204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.693419 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:26.693488 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:26.693853 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 19:05:26.693966 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 19:05:26.694605 8319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.684599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.0.0.1:40904)
0620 19:05:26.685007 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:26.685020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:26.685166 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:26.685534 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:26.685536 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:26.685849 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:26.685950 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.685956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.688067 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.688071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.688983 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.688993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.689560 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.689564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.689698 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.690173 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:26.690188 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:26.690665 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:26.692538 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.692544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.692548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.692758 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.693015 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.693019 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.693021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.693101 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.693556 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:26.693559 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:26.693677 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 19:05:26.694321 (+ 644us) client_negotiation.cc:240] Negotiation successful
0620 19:05:26.694447 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":115,"threads_started":1}
Jun 20 19:05:26 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982325, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.190@KRBTEST.COM for oryx/127.1.154.189@KRBTEST.COM
I20260620 19:05:26.710361 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.700397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52788 (local address 127.1.154.189:45641)
0620 19:05:26.700511 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:26.700516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:26.700785 (+ 269us) server_negotiation.cc:407] Connection header received
0620 19:05:26.701013 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:26.701016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:26.701057 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:26.701144 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:26.702015 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.702774 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.703946 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.704156 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.704265 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.707096 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:26.707100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:26.707102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:26.707132 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:26.708554 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.709029 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.709034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.709036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.709111 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.709387 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.709392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.709393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.709564 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:26.709751 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:26.710083 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 19:05:26.710198 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:05:26.710378 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.700342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:45641 (local address 127.0.0.1:52788)
0620 19:05:26.700668 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:26.700706 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:26.700857 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:26.701181 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:26.701187 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:26.701688 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:26.701817 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.701826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.702895 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.702902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.703825 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.703835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.704266 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.704269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.704374 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.704931 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:26.704944 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:26.706957 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:26.708683 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.708691 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.708698 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.708928 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.709209 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.709212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.709215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.709298 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.709749 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:26.709756 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:26.709880 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 19:05:26.710023 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 19:05:26.710226 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":88,"threads_started":1}
I20260620 19:05:26.712484 7764 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.154.189:45641: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095"
member_type: NON_VOTER
last_known_addr {
host: "127.1.154.189"
port: 45641
}
attrs {
promote: true
}
}
dest_uuid: "eeb660d158aa4c4bb950a43b440f194e"
cas_config_opid_index: -1
I20260620 19:05:26.713263 7764 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: NON_VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: true } }
I20260620 19:05:26.715190 8324 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1a0454ecded94ee4bc4f43bef6153095 (127.1.154.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: NON_VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: true } } }
I20260620 19:05:26.715795 8270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:26.716849 8324 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.154.189:45641
WARNING: no policy specified for oryx/127.1.154.188@KRBTEST.COM; defaulting to no policy
I20260620 19:05:26.717181 8324 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: NON_VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: true } } }
I20260620 19:05:26.717258 8270 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 FOLLOWER]: Refusing update from remote peer eeb660d158aa4c4bb950a43b440f194e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 19:05:26.717276 8324 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:26.717823 8228 master.cc:537] Master@127.1.154.189:45641 shutting down...
I20260620 19:05:26.717985 8325 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: NON_VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:05:26.719609 7751 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 19:05:26.720713 7751 consensus_peers.cc:487] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e -> Peer 1a0454ecded94ee4bc4f43bef6153095 (127.1.154.189:45641): Couldn't send request to peer 1a0454ecded94ee4bc4f43bef6153095. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.154.188@KRBTEST.COM" created.
I20260620 19:05:26.725066 8228 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:26.725360 8228 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:26.725482 8228 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: stopping tablet replica
I20260620 19:05:26.730088 8228 master.cc:552] Master@127.1.154.189:45641 shutdown complete.
I20260620 19:05:26.730202 8228 master_runner.cc:305] Clearing existing system tablet
I20260620 19:05:26.730456 8228 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:26.730528 8228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:26.731602 8228 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:05:26.732422 8330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:26.732601 8228 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:26.732808 8228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:05:26.732851 8228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/wal
uuid: "1a0454ecded94ee4bc4f43bef6153095"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "b728cbd65ec7fa6002fcf363d97b239d"
server_key_iv: "4f992afcb6ec707a02760f69e3db4c3f"
server_key_version: "encryptionkey@0"
I20260620 19:05:26.736177 8228 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:26.740286 8228 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 19:05:26.740464 8228 log.cc:1196] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.154.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.188.keytab.
Entry for principal oryx/127.1.154.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.188.keytab.
I20260620 19:05:26.740798 8228 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: Deleting consensus metadata
I20260620 19:05:26.740998 8228 master_runner.cc:315] Copying system tablet from 127.1.154.190:42505
I20260620 19:05:26.741508 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.188:43659
--webserver_interface=127.1.154.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.188.keytab
--principal=oryx/127.1.154.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:42505,127.1.154.189:45641,127.1.154.188:43659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:05:26.742501 8228 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: tablet copy: Beginning tablet copy session from remote peer at address 127.1.154.190:42505
I20260620 19:05:26.753186 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.743198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.0.0.1:40918)
0620 19:05:26.743506 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:26.743523 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:26.743688 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:26.744116 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:26.744121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:26.744367 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:26.744525 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.744536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.745739 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.745742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.746656 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.746665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.747346 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.747349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.747446 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.748222 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:26.748237 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:26.748667 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:26.750925 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.750929 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.750932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.751193 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.751513 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.751516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.751518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.751605 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.752503 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:26.752506 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:26.752638 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 19:05:26.752789 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 19:05:26.752952 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":85,"threads_started":1}
I20260620 19:05:26.753374 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.743204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.1.154.190:42505)
0620 19:05:26.743356 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:26.743362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:26.743603 (+ 241us) server_negotiation.cc:407] Connection header received
0620 19:05:26.743825 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:26.743830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:26.743889 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:26.743948 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:26.744664 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.745547 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.746797 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.747212 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.747342 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.748894 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:26.748901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:26.748906 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:26.748941 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:26.750801 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.751302 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.751309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.751314 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.751409 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.751707 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.751712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.751714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.752210 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:26.752531 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:26.752897 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 19:05:26.753047 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:05:26.754405 7784 tablet_copy_service.cc:140] P eeb660d158aa4c4bb950a43b440f194e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1a0454ecded94ee4bc4f43bef6153095 ({username='oryx', principal='oryx/127.1.154.189@KRBTEST.COM'} at 127.0.0.1:40918)
I20260620 19:05:26.754637 7784 tablet_copy_service.cc:161] P eeb660d158aa4c4bb950a43b440f194e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1a0454ecded94ee4bc4f43bef6153095 at {username='oryx', principal='oryx/127.1.154.189@KRBTEST.COM'} at 127.0.0.1:40918: session id = 1a0454ecded94ee4bc4f43bef6153095-00000000000000000000000000000000
I20260620 19:05:26.756572 7784 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:05:26.758435 8228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 19:05:26.761688 8228 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: tablet copy: Starting download of 0 data blocks...
I20260620 19:05:26.761876 8228 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: tablet copy: Starting download of 1 WAL segments...
I20260620 19:05:26.763862 8228 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:05:26.766486 7784 tablet_copy_service.cc:342] P eeb660d158aa4c4bb950a43b440f194e: Request end of tablet copy session 1a0454ecded94ee4bc4f43bef6153095-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.154.189@KRBTEST.COM'} at 127.0.0.1:40918
I20260620 19:05:26.766636 7784 tablet_copy_service.cc:433] P eeb660d158aa4c4bb950a43b440f194e: ending tablet copy session 1a0454ecded94ee4bc4f43bef6153095-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1a0454ecded94ee4bc4f43bef6153095
I20260620 19:05:26.775207 8228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:26.777135 8338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:26.777132 8339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:26.777561 8228 server_base.cc:1034] running on GCE node
W20260620 19:05:26.777731 8341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:26.777945 8228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:26.778229 8228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:26.779371 8228 hybrid_clock.cc:648] HybridClock initialized: now 1781982326779357 us; error 31 us; skew 500 ppm
Jun 20 19:05:26 dist-test-slave-6wrf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982326, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:26.780957 8228 init.cc:377] Logged in from keytab as oryx/127.1.154.189@KRBTEST.COM (short username oryx)
I20260620 19:05:26.781421 8228 webserver.cc:460] Webserver started at http://127.1.154.189:40343/ using document root <none> and password file <none>
I20260620 19:05:26.781718 8228 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:26.781795 8228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:26.783188 8228 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:05:26.784466 8348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:26.784746 8228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:05:26.784827 8228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/wal
uuid: "1a0454ecded94ee4bc4f43bef6153095"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "b728cbd65ec7fa6002fcf363d97b239d"
server_key_iv: "4f992afcb6ec707a02760f69e3db4c3f"
server_key_version: "encryptionkey@0"
I20260620 19:05:26.784898 8228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:26.798702 8228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:26.801038 8228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:26.806185 8228 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.189:45641
I20260620 19:05:26.806802 8228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-1/data/info.pb
I20260620 19:05:26.807147 8400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.189:45641 every 8 connection(s)
I20260620 19:05:26.808895 8401 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:05:26.810688 8401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: Bootstrap starting.
I20260620 19:05:26.824088 8401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:05:26.824478 8401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: Bootstrap complete.
I20260620 19:05:26.825212 8401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: NON_VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: true } }
I20260620 19:05:26.825346 8401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1a0454ecded94ee4bc4f43bef6153095, State: Initialized, Role: LEARNER
I20260620 19:05:26.825453 8401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: NON_VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: true } }
I20260620 19:05:26.825868 8404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: NON_VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: true } } }
I20260620 19:05:26.825985 8401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:26.826063 8404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: This master's current role is: LEARNER
I20260620 19:05:26.829535 8415 catalog_manager.cc:1259] Loaded cluster ID: 362b19a36271485bb8eca1f5ab51d9ab
I20260620 19:05:26.829615 8415 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: loading cluster ID for follower catalog manager: success
I20260620 19:05:26.831724 8415 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: acquiring CA information for follower catalog manager: success
I20260620 19:05:26.833081 8415 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 19:05:26.876909 8336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:26.877190 8336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:26.877233 8336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:26.881665 8336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:26.881734 8336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:26.881754 8336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:26.881770 8336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:26.881786 8336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:26.887383 8336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:42505,127.1.154.189:45641,127.1.154.188:43659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.188.keytab
--principal=oryx/127.1.154.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.188:43659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.154.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:26.888501 8336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:26.889511 8336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:26.898450 8423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:26.898474 8420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:26.898450 8421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:26.898890 8336 server_base.cc:1034] running on GCE node
I20260620 19:05:26.899479 8336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:26.900522 8336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:26.901687 8336 hybrid_clock.cc:648] HybridClock initialized: now 1781982326901666 us; error 34 us; skew 500 ppm
Jun 20 19:05:26 dist-test-slave-6wrf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982326, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:26.905727 8336 init.cc:377] Logged in from keytab as oryx/127.1.154.188@KRBTEST.COM (short username oryx)
I20260620 19:05:26.906688 8336 webserver.cc:460] Webserver started at http://127.1.154.188:36493/ using document root <none> and password file <none>
I20260620 19:05:26.907249 8336 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:26.907338 8336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:26.907538 8336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:26.909538 8336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/data/instance:
uuid: "0c97bcc01ac84848bd562c2263093c6a"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "259a897069f2b53088dff3c90ccc7280"
server_key_iv: "ae32d6eac6a911472b264cff5b3efd30"
server_key_version: "encryptionkey@0"
I20260620 19:05:26.910166 8336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/wal/instance:
uuid: "0c97bcc01ac84848bd562c2263093c6a"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "259a897069f2b53088dff3c90ccc7280"
server_key_iv: "ae32d6eac6a911472b264cff5b3efd30"
server_key_version: "encryptionkey@0"
I20260620 19:05:26.914263 8336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:05:26.916817 8430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:26.918386 8336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:26.918501 8336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/wal
uuid: "0c97bcc01ac84848bd562c2263093c6a"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "259a897069f2b53088dff3c90ccc7280"
server_key_iv: "ae32d6eac6a911472b264cff5b3efd30"
server_key_version: "encryptionkey@0"
I20260620 19:05:26.918642 8336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:26.920426 7955 heartbeater.cc:503] Master 127.1.154.190:42505 was elected leader, sending a full tablet report...
I20260620 19:05:26.934139 8336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:26.937395 8336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:26.937680 8336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:26.945585 8336 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.188:43659
I20260620 19:05:26.945936 8482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.188:43659 every 8 connection(s)
I20260620 19:05:26.946678 8336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/data/info.pb
I20260620 19:05:26.949554 8483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:26.952630 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 8336
I20260620 19:05:26.953799 8483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:05:26 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982326, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.188@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
Jun 20 19:05:26 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.154.188@KRBTEST.COM
I20260620 19:05:26.967727 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.956107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40930 (local address 127.1.154.190:42505)
0620 19:05:26.956275 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:26.956280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:26.956555 (+ 275us) server_negotiation.cc:407] Connection header received
0620 19:05:26.956800 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:26.956803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:26.956854 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:26.956948 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:26.957938 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.958454 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.959645 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.959882 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.959958 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.962803 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:26.962807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:26.962810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:26.962850 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:26.964812 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.965388 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.965394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.965399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.965501 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.966203 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.966207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.966209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.966467 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:26.966590 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:26.967411 (+ 821us) server_negotiation.cc:299] Negotiation successful
0620 19:05:26.967536 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 19:05:26.968508 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.954814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.1.154.188:43659)
0620 19:05:26.955384 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:26.955401 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:26.955441 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:05:26.956184 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:26.956213 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:26.956615 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:26.956857 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:26.957639 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.958732 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.959670 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.960071 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.960182 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.963074 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:26.963086 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:26.963096 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:26.963116 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:26.965047 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.965688 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.965696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.965697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.966064 (+ 367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.966434 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.966437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.966438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.967113 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:26.967391 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:26.967723 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 19:05:26.968091 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":217,"server-negotiator.queue_time_us":418,"thread_start_us":186,"threads_started":1}
I20260620 19:05:26.968510 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.955959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.0.0.1:40930)
0620 19:05:26.956408 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:26.956455 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:26.956599 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:26.957006 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:26.957011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:26.957394 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:26.957787 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.957800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.958575 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.958578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.959533 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.959541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.960012 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.960016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.960176 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.960782 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:26.960798 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:26.962414 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:26.964960 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.964968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.964986 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.965270 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.965680 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.965685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.965687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.966067 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.966600 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:26.966609 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:26.967137 (+ 528us) client_negotiation.cc:769] Sending connection context
0620 19:05:26.967354 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 19:05:26.967663 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":88,"spinlock_wait_cycles":12672,"thread_start_us":214,"threads_started":1}
I20260620 19:05:26.970085 8483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:05:26 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982326, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.188@KRBTEST.COM for oryx/127.1.154.189@KRBTEST.COM
I20260620 19:05:26.979823 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.970558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:45641 (local address 127.0.0.1:52804)
0620 19:05:26.970656 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:26.970672 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:26.970752 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:26.971227 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:26.971229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:26.971392 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:26.971476 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.971481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.972345 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.972352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.972887 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.972894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.973336 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.973340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.973440 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.974171 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:26.974188 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:26.975693 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:26.977887 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.977891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.977894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.978139 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.978510 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.978512 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.978514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.978607 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.979107 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:26.979113 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:26.979213 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 19:05:26.979320 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 19:05:26.979651 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 19:05:26.979825 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.970807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52804 (local address 127.1.154.189:45641)
0620 19:05:26.971027 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:26.971030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:26.971041 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:05:26.971075 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:26.971077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:26.971113 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:26.971214 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:26.971591 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.972241 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.973004 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.973232 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.973340 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.975848 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:26.975851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:26.975853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:26.975874 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:26.977716 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.978271 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.978276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.978278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.978367 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.978704 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.978709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.978711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.978975 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:26.979100 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:26.979384 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 19:05:26.979646 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
I20260620 19:05:26.981256 8483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:05:26 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982326, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.188@KRBTEST.COM for oryx/127.1.154.188@KRBTEST.COM
I20260620 19:05:26.991206 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.982020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:43659 (local address 127.0.0.1:48396)
0620 19:05:26.982163 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:26.982181 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:26.982270 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:26.982511 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:26.982514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:26.982677 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:26.982752 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.982757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.983674 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.983676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.984175 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:26.984183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.984647 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.984651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:26.984713 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.985221 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:26.985232 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:26.987055 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:26.989213 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.989216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.989217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.989430 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.989939 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:26.989942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:26.989944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:26.990059 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:26.990504 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:26.990506 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:26.990617 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 19:05:26.990731 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 19:05:26.991000 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 19:05:26.991250 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:26.982120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.1.154.188:43659)
0620 19:05:26.982230 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:26.982234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:26.982246 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:05:26.982351 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:26.982353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:26.982387 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:26.982452 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:26.982871 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.983527 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.984295 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:26.984513 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:26.984598 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:26.987279 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:26.987283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:26.987286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:26.987326 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:26.989075 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.989548 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.989551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.989552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.989708 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:26.990168 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:26.990171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:26.990172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:26.990381 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:26.990469 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:26.990801 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 19:05:26.991067 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":141,"server-negotiator.queue_time_us":48}
I20260620 19:05:26.994745 8483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: Bootstrap starting.
I20260620 19:05:26.997006 8483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:26.997758 8483 log.cc:824] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:26.999668 8483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: No bootstrap required, opened a new log
I20260620 19:05:27.002681 8483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } }
I20260620 19:05:27.002905 8483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:27.002938 8483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c97bcc01ac84848bd562c2263093c6a, State: Initialized, Role: FOLLOWER
I20260620 19:05:27.003309 8483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } }
I20260620 19:05:27.003747 8489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } } }
I20260620 19:05:27.003885 8489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:05:27.004503 8483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [sys.catalog]: configured and running, proceeding with master startup.
W20260620 19:05:27.009341 8500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:05:27.009434 8500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:05:27.012738 8336 master_runner.cc:428] Detected that this master 0c97bcc01ac84848bd562c2263093c6a is joining an existing cluster
I20260620 19:05:27.012835 8336 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.154.188:43659
I20260620 19:05:27.021763 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 7826
I20260620 19:05:27.028385 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:45115
--local_ip_for_outbound_sockets=127.1.154.129
--tserver_master_addrs=127.1.154.190:42505
--webserver_port=44331
--webserver_interface=127.1.154.129
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.129.keytab
--principal=oryx/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.154.190:42505,127.1.154.189:45641
--tserver_master_addrs=127.1.154.190:42505,127.1.154.189:45641,127.1.154.188:43659 with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:05:27.042273 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.015352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:43659 (local address 127.0.0.1:48412)
0620 19:05:27.016341 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.016354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.016464 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.016745 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.016747 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.016930 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:27.017029 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.017036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.017949 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.017953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.018921 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.018930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.019539 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.019542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.019630 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.020158 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:27.020169 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:27.020517 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:27.040023 (+ 19506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.040041 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.040045 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.040315 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.040681 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.040683 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.040685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.040761 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.041172 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:27.041175 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:27.041506 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.041639 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.041798 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":62,"thread_start_us":88,"threads_started":1}
I20260620 19:05:27.043032 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.015440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.1.154.188:43659)
0620 19:05:27.015536 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.015541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.016414 (+ 873us) server_negotiation.cc:407] Connection header received
0620 19:05:27.016549 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.016551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.016590 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.016633 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.017140 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.017843 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.019041 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.019402 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.019512 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.020645 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.020649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.020652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.020675 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.039882 (+ 19207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.040455 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.040460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.040464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.040570 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.040843 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.040845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.040847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.041064 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.042583 (+ 1519us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.042654 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.042878 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":37}
I20260620 19:05:27.043401 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.015020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.1.154.190:42505)
0620 19:05:27.015150 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.015155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.015745 (+ 590us) server_negotiation.cc:407] Connection header received
0620 19:05:27.015902 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.015908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.015959 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.016012 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.016777 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.017585 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.019849 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.020213 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.020288 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.038914 (+ 18626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.038919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.038923 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.038960 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.040807 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.041341 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.041345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.041347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.041437 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.041937 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.041941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.041943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.042191 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.042292 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.043091 (+ 799us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.043223 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:05:27.043717 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.014952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.0.0.1:40934)
0620 19:05:27.015648 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.015663 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.015771 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.016174 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.016179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.016472 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:27.016638 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.016646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.017761 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.017764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.019720 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.019728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.020439 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.020442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.020532 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.021096 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:27.021111 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:27.038587 (+ 17476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:27.040935 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.040938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.040940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.041212 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.041575 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.041582 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.041586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.041771 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.042358 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:27.042361 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:27.042459 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.042594 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.042791 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":67,"spinlock_wait_cycles":20992,"thread_start_us":98,"threads_started":1}
I20260620 19:05:27.045308 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.015099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52810 (local address 127.1.154.189:45641)
0620 19:05:27.015211 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.015215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.015814 (+ 599us) server_negotiation.cc:407] Connection header received
0620 19:05:27.015920 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.015923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.015976 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.016064 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.016732 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.017576 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.019050 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.019435 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.019542 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.040913 (+ 21371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.040928 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.040932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.040966 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.042802 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.043910 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.043914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.043916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.044008 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.044339 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.044342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.044344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.044575 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.044651 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.044994 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.045131 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:05:27.045938 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.015326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:45641 (local address 127.0.0.1:52810)
0620 19:05:27.015736 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.015747 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.015837 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.016097 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.016101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.016438 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:27.016603 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.016612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.017725 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.017728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.018920 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.018930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.019698 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.019701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.019838 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.038477 (+ 18639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:27.038517 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:27.040701 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:27.042930 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.042934 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.042936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.043190 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.044143 (+ 953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.044146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.044148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.044233 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.044727 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:27.044730 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:27.044815 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.045507 (+ 692us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.045747 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":145,"spinlock_wait_cycles":2944,"thread_start_us":93,"threads_started":1}
Jun 20 19:05:27 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982325, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.190@KRBTEST.COM for oryx/127.1.154.188@KRBTEST.COM
I20260620 19:05:27.057412 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.047836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:43659 (local address 127.0.0.1:48414)
0620 19:05:27.048003 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.048017 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.048113 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.048412 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.048415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.048607 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:27.048711 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.048717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.049818 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.049822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.050693 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.050703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.051276 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.051280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.051373 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.051929 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:27.051945 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:27.053553 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:27.055587 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.055591 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.055594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.055860 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.056176 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.056180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.056182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.056277 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.056808 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:27.056811 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:27.056918 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.057049 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.057208 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 19:05:27.057914 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.047906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48414 (local address 127.1.154.188:43659)
0620 19:05:27.048020 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.048025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.048066 (+ 41us) server_negotiation.cc:407] Connection header received
0620 19:05:27.048236 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.048239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.048295 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.048388 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.048914 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.049703 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.050816 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.051150 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.051254 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.053731 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.053735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.053737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.053770 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.055422 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.055978 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.055989 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.055993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.056078 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.056382 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.056385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.056387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.056598 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.056898 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.057157 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.057417 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:05:27.058889 7764 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.154.188:43659: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a"
member_type: NON_VOTER
last_known_addr {
host: "127.1.154.188"
port: 43659
}
attrs {
promote: true
}
}
dest_uuid: "eeb660d158aa4c4bb950a43b440f194e"
cas_config_opid_index: 5
I20260620 19:05:27.059561 7764 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: NON_VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: true } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } }
I20260620 19:05:27.060978 8325 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0c97bcc01ac84848bd562c2263093c6a (127.1.154.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: NON_VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: true } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } } }
I20260620 19:05:27.062553 8325 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.154.188:43659
I20260620 19:05:27.062741 8325 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: NON_VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: true } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } } }
I20260620 19:05:27.062835 8325 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:27.062824 8457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:27.063405 8336 master.cc:537] Master@127.1.154.188:43659 shutting down...
I20260620 19:05:27.064250 8457 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 1 FOLLOWER]: Refusing update from remote peer eeb660d158aa4c4bb950a43b440f194e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 19:05:27.065337 8324 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:05:27.067574 7748 consensus_peers.cc:487] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e -> Peer 0c97bcc01ac84848bd562c2263093c6a (127.1.154.188:43659): Couldn't send request to peer 0c97bcc01ac84848bd562c2263093c6a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:05:27.070590 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.062246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.1.154.189:45641)
0620 19:05:27.064908 (+ 2662us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.064912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.065069 (+ 157us) server_negotiation.cc:407] Connection header received
0620 19:05:27.065270 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.065274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.065331 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.065392 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:27.065898 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.066688 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.068509 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.069185 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.069297 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.069443 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.070245 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.070326 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2572}
I20260620 19:05:27.070858 8489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:27.071030 8489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } }
I20260620 19:05:27.071295 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.062032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:45641 (local address 127.0.0.1:52826)
0620 19:05:27.065003 (+ 2971us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.065019 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.065147 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.065660 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.065663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.065681 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:27.065780 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.065787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.066836 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.066839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.068371 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.068381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.069331 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.069335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.069425 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.069470 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.071022 (+ 1552us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.071111 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2899}
I20260620 19:05:27.071383 8336 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:27.071662 8336 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:27.071982 8367 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 LEARNER]: Refusing update from remote peer eeb660d158aa4c4bb950a43b440f194e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 19:05:27.072021 8489 leader_election.cc:288] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeb660d158aa4c4bb950a43b440f194e (127.1.154.190:42505), 1a0454ecded94ee4bc4f43bef6153095 (127.1.154.189:45641)
I20260620 19:05:27.072299 8336 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: stopping tablet replica
I20260620 19:05:27.072499 8367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c97bcc01ac84848bd562c2263093c6a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a0454ecded94ee4bc4f43bef6153095" is_pre_election: true
I20260620 19:05:27.072507 7774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c97bcc01ac84848bd562c2263093c6a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb660d158aa4c4bb950a43b440f194e" is_pre_election: true
I20260620 19:05:27.072651 8367 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 LEARNER]: Handling vote request from an unknown peer 0c97bcc01ac84848bd562c2263093c6a
I20260620 19:05:27.072769 8367 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 0c97bcc01ac84848bd562c2263093c6a for term 2 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260620 19:05:27.073181 8431 leader_election.cc:302] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c97bcc01ac84848bd562c2263093c6a; no voters: 1a0454ecded94ee4bc4f43bef6153095, eeb660d158aa4c4bb950a43b440f194e
W20260620 19:05:27.073366 8431 raft_consensus.cc:2670] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260620 19:05:27.073484 8324 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: NON_VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:05:27.075353 8367 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0c97bcc01ac84848bd562c2263093c6a (127.1.154.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: NON_VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: true } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } } }
I20260620 19:05:27.077497 8512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: NON_VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: true } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } } }
I20260620 19:05:27.077668 8512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: This master's current role is: LEARNER
I20260620 19:05:27.078066 8404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeb660d158aa4c4bb950a43b440f194e. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: NON_VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: true } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } } }
I20260620 19:05:27.078189 8404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: This master's current role is: LEARNER
I20260620 19:05:27.078809 8336 master.cc:552] Master@127.1.154.188:43659 shutdown complete.
I20260620 19:05:27.078861 8336 master_runner.cc:305] Clearing existing system tablet
I20260620 19:05:27.079097 8336 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:27.079159 8336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:27.080687 8336 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:05:27.081691 8514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:27.081930 8336 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:27.082055 8336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:05:27.082087 8336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/wal
uuid: "0c97bcc01ac84848bd562c2263093c6a"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "259a897069f2b53088dff3c90ccc7280"
server_key_iv: "ae32d6eac6a911472b264cff5b3efd30"
server_key_version: "encryptionkey@0"
I20260620 19:05:27.084112 8336 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:27.086632 8336 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 19:05:27.086731 8336 log.cc:1196] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 19:05:27.086988 8336 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: Deleting consensus metadata
I20260620 19:05:27.087167 8336 master_runner.cc:315] Copying system tablet from 127.1.154.190:42505
I20260620 19:05:27.088299 8336 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: tablet copy: Beginning tablet copy session from remote peer at address 127.1.154.190:42505
I20260620 19:05:27.095501 8324 raft_consensus.cc:1050] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e: attempting to promote NON_VOTER 1a0454ecded94ee4bc4f43bef6153095 to VOTER
I20260620 19:05:27.096501 8324 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } }
I20260620 19:05:27.097968 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.097528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:43659 (local address 127.0.0.1:48422)
0620 19:05:27.097784 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.097858 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260620 19:05:27.098758 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.098526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:43659 (local address 127.0.0.1:48430)
0620 19:05:27.098649 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.098692 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 19:05:27.098891 7748 consensus_peers.cc:487] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e -> Peer 0c97bcc01ac84848bd562c2263093c6a (127.1.154.188:43659): Couldn't send request to peer 0c97bcc01ac84848bd562c2263093c6a. Status: Network error: Client connection negotiation failed: client connection to 127.1.154.188:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 19:05:27.099010 8367 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 LEARNER]: Refusing update from remote peer eeb660d158aa4c4bb950a43b440f194e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 19:05:27.099426 8510 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:05:27.099767 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.088836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.0.0.1:40950)
0620 19:05:27.089101 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.089118 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.089226 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.089580 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.089583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.089881 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:27.090011 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.090018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.091069 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.091075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.092131 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.092145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.092736 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.092740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.092843 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.093427 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:27.093440 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:27.093890 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:27.096153 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.096156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.096159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.096393 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.096720 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.096723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.096725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.096813 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.097256 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:27.097259 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:27.099003 (+ 1744us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.099449 (+ 446us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.099590 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":98,"threads_started":1}
I20260620 19:05:27.100294 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.089198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40950 (local address 127.1.154.190:42505)
0620 19:05:27.089320 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.089325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.089341 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:05:27.089393 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.089396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.089448 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.089559 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.090134 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.090938 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.092267 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.092608 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.092721 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.094040 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.094045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.094047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.094074 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.096031 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.096518 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.096522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.096524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.096605 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.096910 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.096913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.096915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.097119 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.097186 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.099247 (+ 2061us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.099361 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:05:27.100824 7784 tablet_copy_service.cc:140] P eeb660d158aa4c4bb950a43b440f194e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0c97bcc01ac84848bd562c2263093c6a ({username='oryx', principal='oryx/127.1.154.188@KRBTEST.COM'} at 127.0.0.1:40950)
I20260620 19:05:27.100910 7784 tablet_copy_service.cc:161] P eeb660d158aa4c4bb950a43b440f194e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0c97bcc01ac84848bd562c2263093c6a at {username='oryx', principal='oryx/127.1.154.188@KRBTEST.COM'} at 127.0.0.1:40950: session id = 0c97bcc01ac84848bd562c2263093c6a-00000000000000000000000000000000
I20260620 19:05:27.101342 8510 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1a0454ecded94ee4bc4f43bef6153095 (127.1.154.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } } }
I20260620 19:05:27.102151 8367 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1a0454ecded94ee4bc4f43bef6153095 (127.1.154.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } } }
I20260620 19:05:27.103709 8404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } } }
I20260620 19:05:27.103828 8404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:05:27.105334 8510 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } } }
I20260620 19:05:27.105434 8510 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:27.105535 8521 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } } }
I20260620 19:05:27.105718 8521 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:27.106292 7784 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:05:27.107596 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 19:05:27.110002 8336 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: tablet copy: Starting download of 0 data blocks...
I20260620 19:05:27.110209 8336 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: tablet copy: Starting download of 1 WAL segments...
I20260620 19:05:27.111724 8336 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:05:27.115371 7784 tablet_copy_service.cc:342] P eeb660d158aa4c4bb950a43b440f194e: Request end of tablet copy session 0c97bcc01ac84848bd562c2263093c6a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.154.188@KRBTEST.COM'} at 127.0.0.1:40950
I20260620 19:05:27.115479 7784 tablet_copy_service.cc:433] P eeb660d158aa4c4bb950a43b440f194e: ending tablet copy session 0c97bcc01ac84848bd562c2263093c6a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0c97bcc01ac84848bd562c2263093c6a
I20260620 19:05:27.124775 8336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:27.127005 8522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:27.131284 8525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:27.133805 8523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:27.138700 8336 server_base.cc:1034] running on GCE node
I20260620 19:05:27.138932 8336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:27.139264 8336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:27.140396 8336 hybrid_clock.cc:648] HybridClock initialized: now 1781982327140380 us; error 40 us; skew 500 ppm
Jun 20 19:05:27 dist-test-slave-6wrf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982327, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:27.142007 8336 init.cc:377] Logged in from keytab as oryx/127.1.154.188@KRBTEST.COM (short username oryx)
I20260620 19:05:27.142573 8336 webserver.cc:460] Webserver started at http://127.1.154.188:40435/ using document root <none> and password file <none>
I20260620 19:05:27.142779 8336 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:27.142827 8336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:27.144325 8336 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:05:27.145438 8532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:27.145809 8336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:05:27.145888 8336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/wal
uuid: "0c97bcc01ac84848bd562c2263093c6a"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "259a897069f2b53088dff3c90ccc7280"
server_key_iv: "ae32d6eac6a911472b264cff5b3efd30"
server_key_version: "encryptionkey@0"
I20260620 19:05:27.145975 8336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:27.155164 8089 heartbeater.cc:503] Master 127.1.154.190:42505 was elected leader, sending a full tablet report...
I20260620 19:05:27.162846 8336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:27.165050 8336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:27.170476 8336 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.188:43659
I20260620 19:05:27.170821 8584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.188:43659 every 8 connection(s)
I20260620 19:05:27.171084 8336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/master-2/data/info.pb
I20260620 19:05:27.173368 8585 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:05:27.174729 8585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: Bootstrap starting.
W20260620 19:05:27.181768 8509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:27.182063 8509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:27.182132 8509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:27.186367 8509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:27.186439 8509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:27.186540 8509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:05:27.189803 8585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:05:27.190267 8585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: Bootstrap complete.
I20260620 19:05:27.191310 8585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } }
I20260620 19:05:27.191511 8585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0c97bcc01ac84848bd562c2263093c6a, State: Initialized, Role: LEARNER
I20260620 19:05:27.191653 8585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } }
I20260620 19:05:27.192090 8509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.129.keytab
--principal=oryx/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:45115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=44331
--tserver_master_addrs=127.1.154.190:42505,127.1.154.189:45641,127.1.154.188:43659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:27.192189 8591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } } }
I20260620 19:05:27.192480 8585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:27.192660 8591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [sys.catalog]: This master's current role is: LEARNER
I20260620 19:05:27.193253 8509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:27.194406 8509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:05:27.196518 8602 catalog_manager.cc:1259] Loaded cluster ID: 362b19a36271485bb8eca1f5ab51d9ab
I20260620 19:05:27.196637 8602 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: loading cluster ID for follower catalog manager: success
I20260620 19:05:27.198804 8602 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: acquiring CA information for follower catalog manager: success
I20260620 19:05:27.200052 8602 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 19:05:27.206254 8607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:27.206199 8605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:27.206235 8604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:27.206729 8509 server_base.cc:1034] running on GCE node
I20260620 19:05:27.207533 8509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:27.208465 8509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:27.209726 8509 hybrid_clock.cc:648] HybridClock initialized: now 1781982327209702 us; error 45 us; skew 500 ppm
Jun 20 19:05:27 dist-test-slave-6wrf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982327, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:27.212790 8509 init.cc:377] Logged in from keytab as oryx/127.1.154.129@KRBTEST.COM (short username oryx)
I20260620 19:05:27.213918 8509 webserver.cc:460] Webserver started at http://127.1.154.129:44331/ using document root <none> and password file <none>
I20260620 19:05:27.214478 8509 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:27.214584 8509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:27.219029 8509 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:05:27.221975 8614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:27.223531 8509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:27.223692 8509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "70b757ebb93849a9836c58f9969cccbc"
format_stamp: "Formatted at 2026-06-20 19:05:25 on dist-test-slave-6wrf"
server_key: "5f5047162a9d75706cd36949157548b3"
server_key_iv: "8e08bcf5c69e93a7185dce340c759f83"
server_key_version: "encryptionkey@0"
I20260620 19:05:27.224251 8509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:27.237143 8509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:27.241109 8509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:27.241434 8509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:27.242208 8509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:27.243167 8509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:27.243242 8509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:27.243309 8509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:27.243338 8509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:27.252116 8509 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:45115
I20260620 19:05:27.252184 8727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:45115 every 8 connection(s)
I20260620 19:05:27.253407 8509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:05:27.262249 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 8509
I20260620 19:05:27.262462 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 7960
Jun 20 19:05:27 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982327, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.188@KRBTEST.COM
Jun 20 19:05:27 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982327, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
I20260620 19:05:27.269658 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:41521
--local_ip_for_outbound_sockets=127.1.154.130
--tserver_master_addrs=127.1.154.190:42505
--webserver_port=45687
--webserver_interface=127.1.154.130
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.130.keytab
--principal=oryx/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.154.190:42505,127.1.154.189:45641
--tserver_master_addrs=127.1.154.190:42505,127.1.154.189:45641,127.1.154.188:43659 with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:05:27.270263 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.257200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:35353 (local address 127.1.154.190:42505)
0620 19:05:27.257322 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.257326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.257564 (+ 238us) server_negotiation.cc:407] Connection header received
0620 19:05:27.257970 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.257973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.258029 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.258119 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.259671 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.260467 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.261518 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.261876 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.261987 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.265920 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.265925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.265931 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.265964 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.267925 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.268385 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.268388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.268390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.268469 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.268720 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.268725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.268727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.268900 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.269003 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.269980 (+ 977us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.270111 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:05:27.271099 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.257056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.129:35353)
0620 19:05:27.257442 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.257454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.257847 (+ 393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.258437 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.258438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.258822 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:27.259520 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.259529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.260599 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.260602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.261401 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.261409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.262145 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.262149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.262472 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.263445 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:27.263460 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:27.265716 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:27.268034 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.268037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.268040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.268291 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.268552 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.268556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.268558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.268635 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.269040 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:27.269042 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:27.269649 (+ 607us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.270596 (+ 947us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.270735 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":229,"spinlock_wait_cycles":35712,"thread_start_us":100,"threads_started":1}
I20260620 19:05:27.271515 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.255447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:57899 (local address 127.1.154.188:43659)
0620 19:05:27.255701 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.255705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.256917 (+ 1212us) server_negotiation.cc:407] Connection header received
0620 19:05:27.257929 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.257932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.258011 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.258099 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.259676 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.260750 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.261803 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.262145 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.262237 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.265913 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.265916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.265918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.265944 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.267531 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.268079 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.268083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.268085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.268159 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.268601 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.268607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.268611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.268862 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.268969 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.271216 (+ 2247us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.271355 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
Jun 20 19:05:27 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982327, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.189@KRBTEST.COM
I20260620 19:05:27.271109 8734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.255894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:43659 (local address 127.1.154.129:57899)
0620 19:05:27.256834 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.256847 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.257624 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.258293 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.258300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.258870 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:27.259519 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.259529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.260852 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.260855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.261706 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.261717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.262375 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.262380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.262492 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.263267 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:27.263282 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:27.265716 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:27.267667 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.267673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.267688 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.267948 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.268264 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.268270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.268274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.268477 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.268981 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:27.268987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:27.269391 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.269626 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.270028 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":184,"spinlock_wait_cycles":103040,"thread_start_us":171,"threads_started":1}
I20260620 19:05:27.272622 8729 heartbeater.cc:347] Connected to a master server at 127.1.154.190:42505
I20260620 19:05:27.272866 8729 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:27.273401 8728 heartbeater.cc:347] Connected to a master server at 127.1.154.188:43659
I20260620 19:05:27.273444 8729 heartbeater.cc:511] Master 127.1.154.190:42505 requested a full tablet report, sending...
I20260620 19:05:27.273489 8728 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:27.273710 8728 heartbeater.cc:511] Master 127.1.154.188:43659 requested a full tablet report, sending...
I20260620 19:05:27.274657 7764 ts_manager.cc:194] Re-registered known tserver with Master: 70b757ebb93849a9836c58f9969cccbc (127.1.154.129:45115)
I20260620 19:05:27.275434 7764 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:35353
I20260620 19:05:27.275616 8548 ts_manager.cc:194] Registered new tserver with Master: 70b757ebb93849a9836c58f9969cccbc (127.1.154.129:45115)
I20260620 19:05:27.276577 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.255524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:42637 (local address 127.1.154.189:45641)
0620 19:05:27.255653 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.255658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.256597 (+ 939us) server_negotiation.cc:407] Connection header received
0620 19:05:27.257852 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.257855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.257915 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.257983 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.259647 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.260219 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.261728 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.262026 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.262104 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.272112 (+ 10008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.272120 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.272124 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.272158 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.274730 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.275190 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.275194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.275195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.275273 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.275631 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.275635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.275637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.275854 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.275924 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.276317 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.276432 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:05:27.277217 8733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.255894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:45641 (local address 127.1.154.129:42637)
0620 19:05:27.256450 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.256489 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.257624 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.258293 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.258300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.258865 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:27.259510 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.259524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.260445 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.260447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.261564 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.261574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.262271 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.262277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.262450 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.269639 (+ 7189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:27.269654 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:27.271935 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:27.274836 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.274839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.274842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.275070 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.275374 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.275376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.275378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.275459 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.276057 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:27.276060 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:27.276141 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.276906 (+ 765us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.277050 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":100,"spinlock_wait_cycles":17664,"thread_start_us":684,"threads_started":1}
I20260620 19:05:27.278098 8730 heartbeater.cc:347] Connected to a master server at 127.1.154.189:45641
I20260620 19:05:27.278190 8730 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:27.278350 8730 heartbeater.cc:511] Master 127.1.154.189:45641 requested a full tablet report, sending...
I20260620 19:05:27.279301 8357 ts_manager.cc:194] Registered new tserver with Master: 70b757ebb93849a9836c58f9969cccbc (127.1.154.129:45115)
W20260620 19:05:27.396510 8736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:27.396772 8736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:27.396839 8736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 19:05:27.397099 8223 heartbeater.cc:503] Master 127.1.154.190:42505 was elected leader, sending a full tablet report...
W20260620 19:05:27.401479 8736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:27.401566 8736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:27.401675 8736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:27.406838 8736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.130.keytab
--principal=oryx/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:41521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=45687
--tserver_master_addrs=127.1.154.190:42505,127.1.154.189:45641,127.1.154.188:43659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:27.407855 8736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:27.408829 8736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:05:27.418648 8736 server_base.cc:1034] running on GCE node
W20260620 19:05:27.418730 8741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:27.418653 8742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:27.418859 8744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:27.419225 8736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:27.419842 8736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:27.421025 8736 hybrid_clock.cc:648] HybridClock initialized: now 1781982327421011 us; error 28 us; skew 500 ppm
Jun 20 19:05:27 dist-test-slave-6wrf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982327, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:27.424484 8736 init.cc:377] Logged in from keytab as oryx/127.1.154.130@KRBTEST.COM (short username oryx)
I20260620 19:05:27.425508 8736 webserver.cc:460] Webserver started at http://127.1.154.130:45687/ using document root <none> and password file <none>
I20260620 19:05:27.426167 8736 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:27.426249 8736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:27.430744 8736 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 19:05:27.432829 8751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:27.434053 8736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:05:27.434175 8736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "d9095510e212429c884037c06eb9b368"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "65c678e0c3ed0f06e0746b57fb27413a"
server_key_iv: "dda1b8c300e01bff3f91666ca3dec513"
server_key_version: "encryptionkey@0"
I20260620 19:05:27.434722 8736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:27.449680 8736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:27.453280 8736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:27.453500 8736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:27.454116 8736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:27.454969 8736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:27.455036 8736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:27.455114 8736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:27.455160 8736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:27.465047 8736 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:41521
I20260620 19:05:27.465063 8864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:41521 every 8 connection(s)
I20260620 19:05:27.466034 8736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:27.467936 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 8736
I20260620 19:05:27.468144 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 8094
I20260620 19:05:27.475817 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:43351
--local_ip_for_outbound_sockets=127.1.154.131
--tserver_master_addrs=127.1.154.190:42505
--webserver_port=32955
--webserver_interface=127.1.154.131
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.131.keytab
--principal=oryx/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.154.190:42505,127.1.154.189:45641
--tserver_master_addrs=127.1.154.190:42505,127.1.154.189:45641,127.1.154.188:43659 with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:05:27 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982327, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.189@KRBTEST.COM
Jun 20 19:05:27 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982327, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
Jun 20 19:05:27 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982327, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.188@KRBTEST.COM
I20260620 19:05:27.485697 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.467815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:35483 (local address 127.1.154.190:42505)
0620 19:05:27.467966 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.467971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.470948 (+ 2977us) server_negotiation.cc:407] Connection header received
0620 19:05:27.472316 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.472318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.472367 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.472480 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.474112 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.474888 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.476607 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.476939 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.477003 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.481417 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.481421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.481424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.481453 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.482999 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.483726 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.483731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.483735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.483816 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.484264 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.484269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.484272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.484507 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.484638 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.485352 (+ 714us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.485481 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:05:27.487221 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.468277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:60181 (local address 127.1.154.189:45641)
0620 19:05:27.468490 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.468497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.472599 (+ 4102us) server_negotiation.cc:407] Connection header received
0620 19:05:27.472784 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.472788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.472865 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.472952 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.474156 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.474995 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.476630 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.477039 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.477130 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.483436 (+ 6306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.483442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.483447 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.483478 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.485349 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.485917 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.485929 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.485931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.486055 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.486341 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.486344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.486346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.486557 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.486643 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.486946 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.487082 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 19:05:27.487624 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.468249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.130:35483)
0620 19:05:27.470784 (+ 2535us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.470840 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.472085 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.472660 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.472668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.473232 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:27.473928 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.473950 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.475066 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.475069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.476483 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.476493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.477204 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.477209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.477404 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.478407 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:27.478423 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:27.481243 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:27.483120 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.483125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.483145 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.483602 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.483927 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.483931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.483934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.484150 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.484702 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:27.484711 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:27.485057 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.486130 (+ 1073us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.486492 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2291,"mutex_wait_us":308,"spinlock_wait_cycles":166272,"thread_start_us":608,"threads_started":1}
I20260620 19:05:27.488480 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.468249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:45641 (local address 127.1.154.130:60181)
0620 19:05:27.472511 (+ 4262us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.472527 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.472668 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.472977 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.472982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.473293 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:27.473928 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.474027 (+ 99us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.475273 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.475279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.476491 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.476505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.477311 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.477315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.477410 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.478349 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:27.478377 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:27.483292 (+ 4915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:27.485453 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.485457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.485459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.485797 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.486161 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.486163 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.486165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.486245 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.486698 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:27.486701 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:27.486779 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.487565 (+ 786us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.488216 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4151,"mutex_wait_us":1579,"spinlock_wait_cycles":367360,"thread_start_us":115,"threads_started":1}
I20260620 19:05:27.489037 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.470080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:35443 (local address 127.1.154.188:43659)
0620 19:05:27.470204 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.470207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.474207 (+ 4000us) server_negotiation.cc:407] Connection header received
0620 19:05:27.474418 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.474421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.474472 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.474532 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.475062 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.475806 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.476846 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.477174 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.477239 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.483445 (+ 6206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.483452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.483457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.483494 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.485687 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.486346 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.486351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.486354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.486448 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.486811 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.486817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.486821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.487068 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.487146 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.487544 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.488853 (+ 1309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:05:27.489583 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.469740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:43659 (local address 127.1.154.130:35443)
0620 19:05:27.474109 (+ 4369us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.474123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.474310 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.474656 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.474659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.474831 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:27.474933 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.474942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.475937 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.475940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.476738 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.476745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.477387 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.477390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.477531 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.478514 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:27.478541 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:27.483273 (+ 4732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:27.485867 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.485874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.485878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.486218 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.486574 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.486580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.486584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.486676 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.487228 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:27.487231 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:27.487341 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.489236 (+ 1895us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.489388 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4294,"mutex_wait_us":102,"spinlock_wait_cycles":5376,"thread_start_us":89,"threads_started":1}
I20260620 19:05:27.489837 8866 heartbeater.cc:347] Connected to a master server at 127.1.154.190:42505
I20260620 19:05:27.490566 8866 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:27.490695 8865 heartbeater.cc:347] Connected to a master server at 127.1.154.188:43659
I20260620 19:05:27.490830 8865 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:27.490087 8867 heartbeater.cc:347] Connected to a master server at 127.1.154.189:45641
I20260620 19:05:27.491380 8867 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:27.491410 8865 heartbeater.cc:511] Master 127.1.154.188:43659 requested a full tablet report, sending...
I20260620 19:05:27.491636 8867 heartbeater.cc:511] Master 127.1.154.189:45641 requested a full tablet report, sending...
I20260620 19:05:27.492246 8866 heartbeater.cc:511] Master 127.1.154.190:42505 requested a full tablet report, sending...
I20260620 19:05:27.492830 8548 ts_manager.cc:194] Registered new tserver with Master: d9095510e212429c884037c06eb9b368 (127.1.154.130:41521)
I20260620 19:05:27.492915 7764 ts_manager.cc:194] Re-registered known tserver with Master: d9095510e212429c884037c06eb9b368 (127.1.154.130:41521)
I20260620 19:05:27.493702 7764 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:35483
I20260620 19:05:27.493779 8357 ts_manager.cc:194] Registered new tserver with Master: d9095510e212429c884037c06eb9b368 (127.1.154.130:41521)
W20260620 19:05:27.607374 8872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:27.607901 8872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:27.607967 8872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:27.615206 8872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:27.615348 8872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:27.615455 8872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:27.616087 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.611203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:43659 (local address 127.0.0.1:48442)
0620 19:05:27.611525 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.611541 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.611672 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.612161 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.612164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.612183 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:27.612338 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.612350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.613516 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.613519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.614449 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.614459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.615708 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.615711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.615787 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.615811 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.615874 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.615946 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":106,"threads_started":1}
I20260620 19:05:27.616564 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.611368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48442 (local address 127.1.154.188:43659)
0620 19:05:27.611590 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.611597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.611625 (+ 28us) server_negotiation.cc:407] Connection header received
0620 19:05:27.611808 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.611812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.611896 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.612078 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:27.612468 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.613353 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.614578 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.615553 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.615690 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.615910 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.615962 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.616053 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 19:05:27.617264 8553 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 19:05:27.617813 8591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeb660d158aa4c4bb950a43b440f194e. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } } }
I20260620 19:05:27.617954 8591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [sys.catalog]: This master's current role is: LEARNER
I20260620 19:05:27.622927 8872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.131.keytab
--principal=oryx/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:43351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=32955
--tserver_master_addrs=127.1.154.190:42505,127.1.154.189:45641,127.1.154.188:43659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:27.624008 8872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:27.625010 8872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:27.635426 8881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:27.635470 8879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:27.635458 8878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:27.635797 8872 server_base.cc:1034] running on GCE node
I20260620 19:05:27.636406 8872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:27.637321 8872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:27.638536 8872 hybrid_clock.cc:648] HybridClock initialized: now 1781982327638518 us; error 72 us; skew 500 ppm
Jun 20 19:05:27 dist-test-slave-6wrf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982327, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:27.642283 8872 init.cc:377] Logged in from keytab as oryx/127.1.154.131@KRBTEST.COM (short username oryx)
I20260620 19:05:27.643378 8872 webserver.cc:460] Webserver started at http://127.1.154.131:32955/ using document root <none> and password file <none>
I20260620 19:05:27.643925 8872 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:27.643999 8872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:27.648451 8872 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 19:05:27.650971 8888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:27.652690 8872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:27.652856 8872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "d6820357b6174d018dad094c0af49d96"
format_stamp: "Formatted at 2026-06-20 19:05:26 on dist-test-slave-6wrf"
server_key: "097589b7725d1b635a4cdc43ef4c3296"
server_key_iv: "6b8a2628e3d74026fc07b558472b1543"
server_key_version: "encryptionkey@0"
I20260620 19:05:27.653457 8872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:27.672461 8872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:27.676823 8872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:27.677004 8872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:27.677563 8872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:27.678509 8872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:27.678561 8872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:27.678602 8872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:27.678617 8872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:27.688879 8872 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:43351
I20260620 19:05:27.688920 9001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:43351 every 8 connection(s)
I20260620 19:05:27.690017 8872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:27.694300 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 8872
Jun 20 19:05:27 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982327, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.188@KRBTEST.COM
Jun 20 19:05:27 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982327, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.189@KRBTEST.COM
Jun 20 19:05:27 dist-test-slave-6wrf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982327, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
I20260620 19:05:27.707201 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.691940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:36027 (local address 127.1.154.188:43659)
0620 19:05:27.692097 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.692102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.693109 (+ 1007us) server_negotiation.cc:407] Connection header received
0620 19:05:27.694410 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.694415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.694476 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.694528 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.696282 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.697056 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.698901 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.699249 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.699330 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.702679 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.702684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.702687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.702715 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.704556 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.705236 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.705240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.705243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.705336 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.705799 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.705803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.705805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.706058 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.706189 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.706839 (+ 650us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.706968 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 19:05:27.708071 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.691973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:37871 (local address 127.1.154.189:45641)
0620 19:05:27.692093 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.692098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.693396 (+ 1298us) server_negotiation.cc:407] Connection header received
0620 19:05:27.694442 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.694447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.694506 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.694566 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.696249 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.697067 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.700452 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.700790 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.700872 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.703384 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.703387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.703390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.703412 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.705517 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.706636 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.706640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.706641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.706731 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.707050 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.707054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.707056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.707296 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.707399 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.707799 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.707906 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:05:27.708900 9006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.692365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:43659 (local address 127.1.154.131:36027)
0620 19:05:27.692911 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.692972 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.694173 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.694942 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.694952 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.695661 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:27.696110 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.696122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.697237 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.697241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.698625 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.698634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.699524 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.699531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.699752 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.700428 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:27.700451 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:27.702471 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:27.704700 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.704706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.704722 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.705103 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.705459 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.705464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.705467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.705681 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.706211 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:27.706221 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:27.706543 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.707424 (+ 881us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.707793 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":31,"spinlock_wait_cycles":2944,"thread_start_us":718,"threads_started":1}
I20260620 19:05:27.709731 9007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.692365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:45641 (local address 127.1.154.131:37871)
0620 19:05:27.693282 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.693295 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.694177 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.694945 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.694953 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.695581 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:27.696107 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.696126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.697378 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.697380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.699574 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.699582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.700978 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.700982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.701086 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.701701 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:27.701715 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:27.703249 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:27.706244 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.706247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.706249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.706513 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.706839 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.706843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.706847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.706937 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.707444 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:27.707448 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:27.707552 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.707648 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.707897 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":1463,"thread_start_us":123,"threads_started":1}
I20260620 19:05:27.710340 9002 heartbeater.cc:347] Connected to a master server at 127.1.154.188:43659
I20260620 19:05:27.710501 9004 heartbeater.cc:347] Connected to a master server at 127.1.154.189:45641
I20260620 19:05:27.710592 9002 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:27.711239 9002 heartbeater.cc:511] Master 127.1.154.188:43659 requested a full tablet report, sending...
I20260620 19:05:27.710597 9004 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:27.711978 9004 heartbeater.cc:511] Master 127.1.154.189:45641 requested a full tablet report, sending...
I20260620 19:05:27.712442 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.693842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:43733 (local address 127.1.154.190:42505)
0620 19:05:27.694011 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.694015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.694031 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:05:27.694496 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.694502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.694553 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.694600 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.696499 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.697234 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.700852 (+ 3618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.701240 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.701628 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.708672 (+ 7044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.708676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.708679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.708709 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.710530 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.711001 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.711005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.711007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.711091 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.711494 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.711498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.711499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.711708 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.711795 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.712126 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.712247 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 19:05:27.712796 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.697218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.1.154.189:45641)
0620 19:05:27.697653 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.697657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.697814 (+ 157us) server_negotiation.cc:407] Connection header received
0620 19:05:27.698260 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.698267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.698313 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.698393 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.699602 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.700373 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.703894 (+ 3521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.704309 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.704412 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.707139 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.707146 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.707149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.707177 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.708891 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.710957 (+ 2066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.710963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.710968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.711056 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.711417 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.711421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.711423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.711661 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.711807 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.712141 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.712459 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":84,"threads_started":1}
I20260620 19:05:27.712935 8548 ts_manager.cc:194] Registered new tserver with Master: d6820357b6174d018dad094c0af49d96 (127.1.154.131:43351)
I20260620 19:05:27.713145 8357 ts_manager.cc:194] Registered new tserver with Master: d6820357b6174d018dad094c0af49d96 (127.1.154.131:43351)
I20260620 19:05:27.714198 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.693413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.131:43733)
0620 19:05:27.693682 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.693714 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.694173 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.694942 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.694952 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.696277 (+ 1325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:27.696375 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.696380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.697367 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.697371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.700697 (+ 3326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.700709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.701808 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.701813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.701917 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.702408 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:27.702426 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:27.708506 (+ 6080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:27.710665 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.710670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.710674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.710901 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.711245 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:27.711251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:27.711256 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:27.711383 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:27.711854 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:27.711857 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:27.711940 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.713722 (+ 1782us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.714009 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":2091,"thread_start_us":74,"threads_started":1}
I20260620 19:05:27.714830 9003 heartbeater.cc:347] Connected to a master server at 127.1.154.190:42505
I20260620 19:05:27.714931 9003 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:27.715094 9003 heartbeater.cc:511] Master 127.1.154.190:42505 requested a full tablet report, sending...
I20260620 19:05:27.715605 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.697959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48446 (local address 127.1.154.188:43659)
0620 19:05:27.699255 (+ 1296us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.699259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.699272 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:05:27.699306 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.699312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.699364 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.699443 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.700270 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.701001 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.702232 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.702579 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.702684 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.705970 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.705976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.705982 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.706012 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.709949 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.714074 (+ 4125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.714078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.714080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.714221 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.714571 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.714576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.714578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.714834 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.714988 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.715244 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.715393 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1225,"spinlock_wait_cycles":7040,"thread_start_us":76,"threads_started":1}
I20260620 19:05:27.715642 7764 ts_manager.cc:194] Re-registered known tserver with Master: d6820357b6174d018dad094c0af49d96 (127.1.154.131:43351)
I20260620 19:05:27.716360 7764 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:43733
I20260620 19:05:27.719244 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.697225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.1.154.190:42505)
0620 19:05:27.698179 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.698184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.698200 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:05:27.698236 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.698239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.698284 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.698385 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:27.703969 (+ 5584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.704757 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.710086 (+ 5329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.713256 (+ 3170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.713369 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.715962 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:27.715965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:27.715968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:27.715990 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:27.717675 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.718126 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.718128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.718129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.718205 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:27.718525 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:27.718528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:27.718529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:27.718695 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:27.718816 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.719020 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.719118 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":81,"threads_started":1}
I20260620 19:05:27.722550 7764 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:05:27.725518 7764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:27.727975 8324 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: NON_VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: true } } }
I20260620 19:05:27.728152 8324 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:27.731485 8324 raft_consensus.cc:1050] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e: attempting to promote NON_VOTER 0c97bcc01ac84848bd562c2263093c6a to VOTER
I20260620 19:05:27.731779 8324 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } }
I20260620 19:05:27.733115 8553 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 1 LEARNER]: Refusing update from remote peer eeb660d158aa4c4bb950a43b440f194e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 19:05:27.734015 8367 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 FOLLOWER]: Refusing update from remote peer eeb660d158aa4c4bb950a43b440f194e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 19:05:27.734470 9021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 19:05:27.735122 9020 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 19:05:27.736250 9020 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0c97bcc01ac84848bd562c2263093c6a (127.1.154.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } } }
I20260620 19:05:27.736707 8553 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0c97bcc01ac84848bd562c2263093c6a (127.1.154.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } } }
I20260620 19:05:27.736938 8367 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0c97bcc01ac84848bd562c2263093c6a (127.1.154.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } } }
I20260620 19:05:27.738528 8324 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } } }
I20260620 19:05:27.738633 8324 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:27.738600 8591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } } }
I20260620 19:05:27.738711 8591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:05:27.739326 9021 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } } }
I20260620 19:05:27.739436 9021 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:27.739384 8324 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } } }
I20260620 19:05:27.739491 8324 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:27.740713 9029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eeb660d158aa4c4bb950a43b440f194e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } } }
I20260620 19:05:27.740968 9029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:05:27.747067 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.741569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:43351 (local address 127.0.0.1:53860)
0620 19:05:27.741695 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.741711 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.741794 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.743084 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.743086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.743096 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:27.743178 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.743183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.744614 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.744617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.745786 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.745792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.746675 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.746679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.746768 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.746789 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.746849 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.746926 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":35}
I20260620 19:05:27.747110 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.742378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.1.154.131:43351)
0620 19:05:27.742656 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.742664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.742683 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:05:27.742744 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.742748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.742919 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.743124 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:27.743288 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.744479 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.745894 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.746560 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.746678 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.746831 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.746893 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.746980 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
I20260620 19:05:27.749157 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.741569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:45115 (local address 127.0.0.1:33614)
0620 19:05:27.742889 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.742903 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.742994 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.743402 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.743404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.743418 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:27.743476 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.743480 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.744748 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.744758 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.746109 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.746119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.748345 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.748349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.748641 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.748673 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.748748 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.748856 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1263,"mutex_wait_us":1141,"thread_start_us":118,"threads_started":1}
I20260620 19:05:27.750237 9032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.741823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.1.154.129:45115)
0620 19:05:27.742461 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.742475 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.743006 (+ 531us) server_negotiation.cc:407] Connection header received
0620 19:05:27.743087 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.743092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.743279 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.743419 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:27.743555 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.744620 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.746576 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.747813 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.748061 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.748236 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.749742 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.750053 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":96,"threads_started":1}
I20260620 19:05:27.751160 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.741803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:41521 (local address 127.0.0.1:51020)
0620 19:05:27.742007 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.742020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.742101 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.745397 (+ 3296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.745400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.745417 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:27.745517 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.745523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.747513 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.747517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.748400 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.748411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.750744 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.750749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.750851 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.750880 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.750939 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.751015 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":27,"thread_start_us":92,"threads_started":1}
I20260620 19:05:27.751798 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.742361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51020 (local address 127.1.154.130:41521)
0620 19:05:27.744985 (+ 2624us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.744995 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.745017 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:05:27.745084 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.745090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.745263 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.745400 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:27.745671 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.747143 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.749382 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.750561 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.751430 (+ 869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.751552 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.751596 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.751684 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2531,"thread_start_us":102,"threads_started":1}
I20260620 19:05:27.752278 8936 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6ea90949a14d35a94c3545cd85f346 (DEFAULT_TABLE table=test-table [id=d16e8ebca8564d52a967b82070fecd10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:27.753675 8936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6ea90949a14d35a94c3545cd85f346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:27.753961 8662 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6ea90949a14d35a94c3545cd85f346 (DEFAULT_TABLE table=test-table [id=d16e8ebca8564d52a967b82070fecd10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:27.754225 8799 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6ea90949a14d35a94c3545cd85f346 (DEFAULT_TABLE table=test-table [id=d16e8ebca8564d52a967b82070fecd10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:27.754992 8662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6ea90949a14d35a94c3545cd85f346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:27.755138 8799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6ea90949a14d35a94c3545cd85f346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:27.762297 9036 tablet_bootstrap.cc:492] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc: Bootstrap starting.
I20260620 19:05:27.762933 9037 tablet_bootstrap.cc:492] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96: Bootstrap starting.
I20260620 19:05:27.765208 9037 tablet_bootstrap.cc:654] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:27.765499 9038 tablet_bootstrap.cc:492] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368: Bootstrap starting.
I20260620 19:05:27.767284 9038 tablet_bootstrap.cc:654] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:27.767421 9036 tablet_bootstrap.cc:654] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:27.768090 9038 log.cc:824] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:27.768090 9036 log.cc:824] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:27.768904 9037 log.cc:824] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:27.770082 9036 tablet_bootstrap.cc:492] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc: No bootstrap required, opened a new log
I20260620 19:05:27.770076 9038 tablet_bootstrap.cc:492] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368: No bootstrap required, opened a new log
I20260620 19:05:27.770268 9038 ts_tablet_manager.cc:1397] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:05:27.770268 9036 ts_tablet_manager.cc:1397] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 19:05:27.771319 9037 tablet_bootstrap.cc:492] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96: No bootstrap required, opened a new log
I20260620 19:05:27.771611 9037 ts_tablet_manager.cc:1397] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260620 19:05:27.774493 9036 raft_consensus.cc:348] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } }
I20260620 19:05:27.774796 9036 raft_consensus.cc:374] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:27.774859 9036 raft_consensus.cc:729] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b757ebb93849a9836c58f9969cccbc, State: Initialized, Role: FOLLOWER
I20260620 19:05:27.774755 9038 raft_consensus.cc:348] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } }
I20260620 19:05:27.774966 9038 raft_consensus.cc:374] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:27.775017 9038 raft_consensus.cc:729] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9095510e212429c884037c06eb9b368, State: Initialized, Role: FOLLOWER
I20260620 19:05:27.775388 9036 consensus_queue.cc:260] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } }
I20260620 19:05:27.775388 9038 consensus_queue.cc:260] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } }
I20260620 19:05:27.776407 9038 ts_tablet_manager.cc:1428] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 19:05:27.776407 9036 ts_tablet_manager.cc:1428] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:05:27.776680 8729 heartbeater.cc:503] Master 127.1.154.190:42505 was elected leader, sending a full tablet report...
I20260620 19:05:27.776602 9037 raft_consensus.cc:348] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } }
I20260620 19:05:27.776948 9037 raft_consensus.cc:374] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:27.777012 9037 raft_consensus.cc:729] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6820357b6174d018dad094c0af49d96, State: Initialized, Role: FOLLOWER
I20260620 19:05:27.778026 9037 consensus_queue.cc:260] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } }
I20260620 19:05:27.779376 9037 ts_tablet_manager.cc:1428] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260620 19:05:27.779984 9003 heartbeater.cc:503] Master 127.1.154.190:42505 was elected leader, sending a full tablet report...
I20260620 19:05:27.783413 8866 heartbeater.cc:503] Master 127.1.154.190:42505 was elected leader, sending a full tablet report...
W20260620 19:05:27.941215 9005 tablet.cc:2369] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:05:27.960486 9044 raft_consensus.cc:482] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:27.960745 9044 raft_consensus.cc:504] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } }
I20260620 19:05:27.962062 9044 leader_election.cc:288] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9095510e212429c884037c06eb9b368 (127.1.154.130:41521), 70b757ebb93849a9836c58f9969cccbc (127.1.154.129:45115)
I20260620 19:05:27.965808 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.962408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:49755 (local address 127.1.154.130:41521)
0620 19:05:27.962546 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.962551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.962568 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:05:27.962638 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.962643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.962690 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.962773 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:27.963126 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.963646 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.964774 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.965162 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.965283 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.965337 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.965562 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.965668 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 19:05:27.965819 9032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.962563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:60033 (local address 127.1.154.129:45115)
0620 19:05:27.962682 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:27.962685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:27.962698 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:05:27.962766 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:27.962771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:27.962817 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:27.962894 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:27.963147 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.963650 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.964749 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.965174 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.965304 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.965368 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:27.965615 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 19:05:27.965693 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:05:27.965844 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.962266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:41521 (local address 127.1.154.131:49755)
0620 19:05:27.962403 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.962420 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.962517 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.962799 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.962802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.962819 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:27.962967 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.962976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.963759 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.963765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.964620 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.964629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.965280 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.965284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.965396 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.965457 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.965524 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.965669 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":129}
I20260620 19:05:27.965845 9007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:27.962438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:45115 (local address 127.1.154.131:60033)
0620 19:05:27.962595 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:27.962608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:27.962675 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:27.962919 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:27.962921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:27.962937 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:27.963016 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.963020 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.963760 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.963765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.964615 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:27.964623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:27.965309 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:27.965313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:27.965417 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:27.965468 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 19:05:27.965557 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 19:05:27.965668 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":50}
I20260620 19:05:27.966491 8682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6ea90949a14d35a94c3545cd85f346" candidate_uuid: "d6820357b6174d018dad094c0af49d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70b757ebb93849a9836c58f9969cccbc" is_pre_election: true
I20260620 19:05:27.966500 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6ea90949a14d35a94c3545cd85f346" candidate_uuid: "d6820357b6174d018dad094c0af49d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9095510e212429c884037c06eb9b368" is_pre_election: true
I20260620 19:05:27.966795 8682 raft_consensus.cc:2454] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6820357b6174d018dad094c0af49d96 in term 0.
I20260620 19:05:27.966801 8819 raft_consensus.cc:2454] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6820357b6174d018dad094c0af49d96 in term 0.
I20260620 19:05:27.967250 8892 leader_election.cc:302] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6820357b6174d018dad094c0af49d96, d9095510e212429c884037c06eb9b368; no voters:
I20260620 19:05:27.967509 9044 raft_consensus.cc:2781] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:27.967582 9044 raft_consensus.cc:482] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:27.967613 9044 raft_consensus.cc:3037] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:05:27.967886 8868 tablet.cc:2369] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:05:27.968706 9044 raft_consensus.cc:504] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } }
I20260620 19:05:27.969067 9044 leader_election.cc:288] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [CANDIDATE]: Term 1 election: Requested vote from peers d9095510e212429c884037c06eb9b368 (127.1.154.130:41521), 70b757ebb93849a9836c58f9969cccbc (127.1.154.129:45115)
I20260620 19:05:27.969370 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6ea90949a14d35a94c3545cd85f346" candidate_uuid: "d6820357b6174d018dad094c0af49d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9095510e212429c884037c06eb9b368"
I20260620 19:05:27.969462 8682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6ea90949a14d35a94c3545cd85f346" candidate_uuid: "d6820357b6174d018dad094c0af49d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70b757ebb93849a9836c58f9969cccbc"
I20260620 19:05:27.969507 8819 raft_consensus.cc:3037] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:27.969645 8682 raft_consensus.cc:3037] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:27.970552 8819 raft_consensus.cc:2454] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6820357b6174d018dad094c0af49d96 in term 1.
I20260620 19:05:27.970656 8682 raft_consensus.cc:2454] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6820357b6174d018dad094c0af49d96 in term 1.
I20260620 19:05:27.970805 8892 leader_election.cc:302] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6820357b6174d018dad094c0af49d96, d9095510e212429c884037c06eb9b368; no voters:
I20260620 19:05:27.970971 9044 raft_consensus.cc:2781] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:27.971174 9044 raft_consensus.cc:686] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [term 1 LEADER]: Becoming Leader. State: Replica: d6820357b6174d018dad094c0af49d96, State: Running, Role: LEADER
I20260620 19:05:27.971508 9044 consensus_queue.cc:237] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } }
I20260620 19:05:27.974781 7764 catalog_manager.cc:5495] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6820357b6174d018dad094c0af49d96 (127.1.154.131). New cstate: current_term: 1 leader_uuid: "d6820357b6174d018dad094c0af49d96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } health_report { overall_health: UNKNOWN } } }
W20260620 19:05:28.005764 8731 tablet.cc:2369] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:05:28.046416 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:28.042459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.1.154.131:43351)
0620 19:05:28.042622 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:28.042626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:28.042642 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:05:28.042722 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:28.042724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:28.042782 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:28.042878 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:28.043143 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:28.043669 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:28.044532 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:28.044748 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:28.044839 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:28.045387 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:28.045885 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:28.046031 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:28.046227 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 19:05:28.046279 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 19:05:28.054705 8819 raft_consensus.cc:1261] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368 [term 1 FOLLOWER]: Refusing update from remote peer d6820357b6174d018dad094c0af49d96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:28.054721 8682 raft_consensus.cc:1261] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc [term 1 FOLLOWER]: Refusing update from remote peer d6820357b6174d018dad094c0af49d96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:28.055541 9045 consensus_queue.cc:1035] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:28.055744 9044 consensus_queue.cc:1035] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:28.063390 9048 mvcc.cc:205] Tried to move back new op lower bound from 7298999615702945792 to 7298999615373803520. Current Snapshot: MvccSnapshot[applied={T|T < 7298999615702945792}]
I20260620 19:05:28.066041 9049 mvcc.cc:205] Tried to move back new op lower bound from 7298999615702945792 to 7298999615373803520. Current Snapshot: MvccSnapshot[applied={T|T < 7298999615702945792}]
I20260620 19:05:28.069388 9050 mvcc.cc:205] Tried to move back new op lower bound from 7298999615702945792 to 7298999615373803520. Current Snapshot: MvccSnapshot[applied={T|T < 7298999615702945792}]
I20260620 19:05:28.075408 7760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:28.075626 7760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:28.081015 7760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eeb660d158aa4c4bb950a43b440f194e: Sending DeleteTablet for 3 replicas of tablet 9c6ea90949a14d35a94c3545cd85f346
I20260620 19:05:28.081804 8936 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6ea90949a14d35a94c3545cd85f346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:28 UTC) from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:53860
I20260620 19:05:28.081888 8799 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6ea90949a14d35a94c3545cd85f346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:28 UTC) from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:51020
I20260620 19:05:28.082013 8662 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6ea90949a14d35a94c3545cd85f346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:28 UTC) from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:33614
I20260620 19:05:28.082305 9057 tablet_replica.cc:330] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96: stopping tablet replica
I20260620 19:05:28.082615 9057 raft_consensus.cc:2229] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:05:28.083026 9057 raft_consensus.cc:2258] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:28.083701 9059 tablet_replica.cc:330] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc: stopping tablet replica
I20260620 19:05:28.083968 9059 raft_consensus.cc:2229] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:28.084283 9059 raft_consensus.cc:2258] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:28.084566 9057 ts_tablet_manager.cc:1905] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:28.085872 9059 ts_tablet_manager.cc:1905] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:28.086100 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 7732
I20260620 19:05:28.088033 9057 ts_tablet_manager.cc:1918] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:28.088150 9057 log.cc:1196] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/9c6ea90949a14d35a94c3545cd85f346
I20260620 19:05:28.088480 9057 ts_tablet_manager.cc:1939] T 9c6ea90949a14d35a94c3545cd85f346 P d6820357b6174d018dad094c0af49d96: Deleting consensus metadata
I20260620 19:05:28.089982 9059 ts_tablet_manager.cc:1918] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:28.090088 9059 log.cc:1196] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-0/wal/wals/9c6ea90949a14d35a94c3545cd85f346
I20260620 19:05:28.090395 9059 ts_tablet_manager.cc:1939] T 9c6ea90949a14d35a94c3545cd85f346 P 70b757ebb93849a9836c58f9969cccbc: Deleting consensus metadata
I20260620 19:05:28.084600 9058 tablet_replica.cc:330] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368: stopping tablet replica
I20260620 19:05:28.091786 9058 raft_consensus.cc:2229] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:28.092123 9058 raft_consensus.cc:2258] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 19:05:28.093622 8889 connection.cc:535] server connection from 127.0.0.1:53860 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:05:28.093886 9058 ts_tablet_manager.cc:1905] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:05:28.094584 8617 connection.cc:535] server connection from 127.0.0.1:33614 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:05:28.097149 9058 ts_tablet_manager.cc:1918] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:28.097249 9058 log.cc:1196] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/9c6ea90949a14d35a94c3545cd85f346
I20260620 19:05:28.097538 9058 ts_tablet_manager.cc:1939] T 9c6ea90949a14d35a94c3545cd85f346 P d9095510e212429c884037c06eb9b368: Deleting consensus metadata
W20260620 19:05:28.098624 8752 connection.cc:441] server connection from 127.0.0.1:51020 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51020 (request call id 1) could send its response
I20260620 19:05:28.103339 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:28.084406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.1.154.189:45641)
0620 19:05:28.084641 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:28.084646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:28.089978 (+ 5332us) server_negotiation.cc:407] Connection header received
0620 19:05:28.090463 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:28.090466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:28.090517 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:28.090569 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:28.091632 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:28.092477 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:28.093885 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:28.094249 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:28.094323 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:28.098811 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:28.098815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:28.098818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:28.098847 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:28.101285 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:28.101785 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:28.101789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:28.101790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:28.101867 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:28.102133 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:28.102136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:28.102138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:28.102342 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:28.102450 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:28.102777 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 19:05:28.102886 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260620 19:05:28.108307 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:28.098980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48464 (local address 127.1.154.188:43659)
0620 19:05:28.099130 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:28.099140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:28.099156 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:05:28.099200 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:28.099203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:28.099251 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:28.099330 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:28.100176 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:28.100948 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:28.102142 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:28.102475 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:28.102544 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:28.104339 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:28.104343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:28.104346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:28.104369 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:28.106523 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:28.107033 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:28.107036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:28.107038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:28.107114 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:28.107396 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:28.107399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:28.107401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:28.107599 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:28.107709 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:28.108046 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 19:05:28.108151 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:05:28.109004 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:28.100009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.1.154.189:45641)
0620 19:05:28.100137 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:28.100142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:28.100155 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:05:28.100186 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:28.100188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:28.100230 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:28.100315 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:28.101417 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:28.102184 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:28.103286 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:28.103581 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:28.103655 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:28.105645 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:28.105650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:28.105655 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:28.105684 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:28.107275 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:28.107713 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:28.107717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:28.107719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:28.107802 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:28.108096 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:28.108099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:28.108101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:28.108274 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:28.108364 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:28.108690 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 19:05:28.108805 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:05:28.113075 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:28.084517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.1.154.188:43659)
0620 19:05:28.084732 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:28.084736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:28.104965 (+ 20229us) server_negotiation.cc:407] Connection header received
0620 19:05:28.105145 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:28.105150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:28.105197 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:28.105254 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:28.106057 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:28.106900 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:28.108021 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:28.108353 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:28.108425 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:28.109995 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:28.109999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:28.110002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:28.110026 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:28.111427 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:28.111832 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:28.111836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:28.111837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:28.111915 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:28.112189 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:28.112192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:28.112194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:28.112388 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:28.112503 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:28.112822 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 19:05:28.112934 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 19:05:29.069517 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.068638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.129:40875)
0620 19:05:29.069289 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.069399 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":265,"threads_started":1}
W20260620 19:05:29.069855 8729 heartbeater.cc:644] Failed to heartbeat to 127.1.154.190:42505 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
I20260620 19:05:29.069970 9068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.069443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.130:56293)
0620 19:05:29.069768 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.069851 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":73,"threads_started":1}
W20260620 19:05:29.070235 8866 heartbeater.cc:644] Failed to heartbeat to 127.1.154.190:42505 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
I20260620 19:05:29.070871 9069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.070506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.131:47597)
0620 19:05:29.070735 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.070799 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":54,"threads_started":1}
W20260620 19:05:29.071169 9003 heartbeater.cc:644] Failed to heartbeat to 127.1.154.190:42505 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
I20260620 19:05:29.581226 9070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeb660d158aa4c4bb950a43b440f194e)
I20260620 19:05:29.581459 9070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } }
I20260620 19:05:29.582082 9070 leader_election.cc:288] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeb660d158aa4c4bb950a43b440f194e (127.1.154.190:42505), 1a0454ecded94ee4bc4f43bef6153095 (127.1.154.189:45641)
I20260620 19:05:29.582583 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.582165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.0.0.1:41434)
0620 19:05:29.582369 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.582476 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
W20260620 19:05:29.583006 8533 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
I20260620 19:05:29.583442 9074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeb660d158aa4c4bb950a43b440f194e)
I20260620 19:05:29.583631 9074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } }
I20260620 19:05:29.584060 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.583840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.0.0.1:41442)
0620 19:05:29.583947 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.584004 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 19:05:29.584224 8533 leader_election.cc:334] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer eeb660d158aa4c4bb950a43b440f194e (127.1.154.190:42505): Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
I20260620 19:05:29.584664 9074 leader_election.cc:288] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeb660d158aa4c4bb950a43b440f194e (127.1.154.190:42505), 0c97bcc01ac84848bd562c2263093c6a (127.1.154.188:43659)
I20260620 19:05:29.585219 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.584791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.0.0.1:41450)
0620 19:05:29.585044 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.585121 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
W20260620 19:05:29.585393 8349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
I20260620 19:05:29.586226 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.586001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.0.0.1:41452)
0620 19:05:29.586086 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.586169 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 19:05:29.586331 8349 leader_election.cc:334] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer eeb660d158aa4c4bb950a43b440f194e (127.1.154.190:42505): Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
I20260620 19:05:29.587265 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.582469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:45641 (local address 127.0.0.1:52864)
0620 19:05:29.582990 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.583017 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:29.583285 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:29.583680 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:29.583683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:29.583704 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:29.583814 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:29.583820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.584901 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.584904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:29.586236 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:29.586243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.586935 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.586941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:29.587045 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:29.587072 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:05:29.587128 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 19:05:29.587180 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":60,"threads_started":1}
I20260620 19:05:29.587340 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.582593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.1.154.189:45641)
0620 19:05:29.583354 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:29.583359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:29.583380 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:05:29.583438 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:29.583441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:29.583491 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:29.583961 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:29.584053 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.584774 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.586379 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.586821 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.586903 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:29.586947 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:29.587156 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 19:05:29.587224 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":314,"threads_started":1}
I20260620 19:05:29.587810 8367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c97bcc01ac84848bd562c2263093c6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1a0454ecded94ee4bc4f43bef6153095" is_pre_election: true
I20260620 19:05:29.587984 8367 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c97bcc01ac84848bd562c2263093c6a in term 1.
I20260620 19:05:29.588301 8536 leader_election.cc:302] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c97bcc01ac84848bd562c2263093c6a, 1a0454ecded94ee4bc4f43bef6153095; no voters: eeb660d158aa4c4bb950a43b440f194e
I20260620 19:05:29.588531 9070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:05:29.588599 9070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeb660d158aa4c4bb950a43b440f194e)
I20260620 19:05:29.588639 9070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:05:29.589465 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.585260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48476 (local address 127.1.154.188:43659)
0620 19:05:29.585536 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:29.585546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:29.585560 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:05:29.585741 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:29.585744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:29.585789 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:29.585904 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:29.586132 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.586910 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.588190 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.588799 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.588930 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:29.588980 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:29.589298 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 19:05:29.589361 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":84,"threads_started":1}
I20260620 19:05:29.589506 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.585124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:43659 (local address 127.0.0.1:48476)
0620 19:05:29.585485 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.585498 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:29.585582 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:29.585930 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:29.585933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:29.585950 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:29.586036 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:29.586042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.587019 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.587022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:29.588058 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:29.588067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.589001 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.589006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:29.589115 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:29.589155 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 19:05:29.589267 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 19:05:29.589354 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":57,"threads_started":1}
I20260620 19:05:29.589949 9070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } }
I20260620 19:05:29.590188 8553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a0454ecded94ee4bc4f43bef6153095" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0c97bcc01ac84848bd562c2263093c6a" is_pre_election: true
I20260620 19:05:29.590314 9070 leader_election.cc:288] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [CANDIDATE]: Term 2 election: Requested vote from peers eeb660d158aa4c4bb950a43b440f194e (127.1.154.190:42505), 1a0454ecded94ee4bc4f43bef6153095 (127.1.154.189:45641)
I20260620 19:05:29.590409 8553 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1a0454ecded94ee4bc4f43bef6153095 in current term 2: Already voted for candidate 0c97bcc01ac84848bd562c2263093c6a in this term.
I20260620 19:05:29.590672 8367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c97bcc01ac84848bd562c2263093c6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1a0454ecded94ee4bc4f43bef6153095"
I20260620 19:05:29.590811 8367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:05:29.590911 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.590629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.0.0.1:41458)
0620 19:05:29.590789 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.590847 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 19:05:29.591584 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.591362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.0.0.1:41462)
0620 19:05:29.591467 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.591515 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 19:05:29.591717 8533 leader_election.cc:334] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer eeb660d158aa4c4bb950a43b440f194e (127.1.154.190:42505): Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
I20260620 19:05:29.591850 8367 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c97bcc01ac84848bd562c2263093c6a in term 2.
I20260620 19:05:29.592087 8536 leader_election.cc:302] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c97bcc01ac84848bd562c2263093c6a, 1a0454ecded94ee4bc4f43bef6153095; no voters: eeb660d158aa4c4bb950a43b440f194e
I20260620 19:05:29.592139 8349 leader_election.cc:302] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a0454ecded94ee4bc4f43bef6153095; no voters: 0c97bcc01ac84848bd562c2263093c6a, eeb660d158aa4c4bb950a43b440f194e
I20260620 19:05:29.592262 9070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:05:29.592356 9074 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:05:29.592494 9070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [term 2 LEADER]: Becoming Leader. State: Replica: 0c97bcc01ac84848bd562c2263093c6a, State: Running, Role: LEADER
I20260620 19:05:29.592711 9070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } }
I20260620 19:05:29.594079 9080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c97bcc01ac84848bd562c2263093c6a. Latest consensus state: current_term: 2 leader_uuid: "0c97bcc01ac84848bd562c2263093c6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } } }
I20260620 19:05:29.594190 9080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:29.594480 9082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:29.596752 9082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:29.597134 9082 catalog_manager.cc:1259] Loaded cluster ID: 362b19a36271485bb8eca1f5ab51d9ab
I20260620 19:05:29.597200 9082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:29.598234 9082 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:29.599088 9082 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: Loaded TSK: 0
I20260620 19:05:29.599803 9082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:05:29.646055 8548 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:05:29.649859 8548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:29.654477 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.654190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.0.0.1:41484)
0620 19:05:29.654324 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.654393 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 19:05:29.654737 8367 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [term 2 FOLLOWER]: Refusing update from remote peer 0c97bcc01ac84848bd562c2263093c6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 19:05:29.655283 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.655060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.0.0.1:41486)
0620 19:05:29.655175 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.655230 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 19:05:29.655368 9080 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 19:05:29.655725 8533 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a -> Peer eeb660d158aa4c4bb950a43b440f194e (127.1.154.190:42505): Couldn't send request to peer eeb660d158aa4c4bb950a43b440f194e. Status: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 19:05:29.657449 9074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c97bcc01ac84848bd562c2263093c6a. Latest consensus state: current_term: 2 leader_uuid: "0c97bcc01ac84848bd562c2263093c6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } } }
I20260620 19:05:29.657647 9074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:05:29.658818 9070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "0c97bcc01ac84848bd562c2263093c6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } } }
I20260620 19:05:29.658957 9070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:29.659250 9074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "0c97bcc01ac84848bd562c2263093c6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eeb660d158aa4c4bb950a43b440f194e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 42505 } } peers { permanent_uuid: "1a0454ecded94ee4bc4f43bef6153095" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 45641 } attrs { promote: false } } peers { permanent_uuid: "0c97bcc01ac84848bd562c2263093c6a" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 43659 } attrs { promote: false } } }
I20260620 19:05:29.659377 9074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0454ecded94ee4bc4f43bef6153095 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:05:29.670667 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.666607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:43351 (local address 127.0.0.1:53874)
0620 19:05:29.666793 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.666809 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:29.666889 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:29.667286 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:29.667290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:29.667308 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:29.667423 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:29.667430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.668422 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.668425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:29.669368 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:29.669376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.670278 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.670282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:29.670368 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:29.670396 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 19:05:29.670444 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 19:05:29.670525 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":21}
I20260620 19:05:29.671380 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.666766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.1.154.131:43351)
0620 19:05:29.667054 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:29.667058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:29.667074 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:05:29.667127 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:29.667130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:29.667177 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:29.667268 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:29.667536 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.668297 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.669500 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.670152 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.671069 (+ 917us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:29.671131 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:29.671167 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 19:05:29.671249 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
I20260620 19:05:29.671654 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.666920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:45115 (local address 127.0.0.1:33620)
0620 19:05:29.667154 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.667170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:29.667239 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:29.667651 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:29.667653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:29.667670 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:29.667768 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:29.667773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.669267 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.669271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:29.670445 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:29.670453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.671282 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.671285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:29.671364 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:29.671395 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 19:05:29.671457 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:05:29.671508 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260620 19:05:29.671684 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.667875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.1.154.130:41521)
0620 19:05:29.668114 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:29.668118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:29.668130 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:05:29.668177 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:29.668180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:29.668240 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:29.668317 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:29.668586 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.669323 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.670391 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.671009 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.671106 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:29.671169 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:29.671443 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 19:05:29.671540 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":101,"threads_started":1}
I20260620 19:05:29.671658 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.667435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:41521 (local address 127.0.0.1:51024)
0620 19:05:29.667660 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.667672 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:29.667751 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:29.668333 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:29.668336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:29.668352 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:29.668458 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:29.668464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.669425 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.669430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:29.670238 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:29.670246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.671215 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.671218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:29.671324 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:29.671349 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 19:05:29.671426 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:05:29.671506 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260620 19:05:29.672021 9089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.667178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33620 (local address 127.1.154.129:45115)
0620 19:05:29.667400 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:29.667410 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:29.667421 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:05:29.667461 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:29.667463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:29.667527 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:29.668246 (+ 719us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:29.668350 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.669141 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.670577 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.671179 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.671279 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:29.671336 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:29.671814 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 19:05:29.671896 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":63,"threads_started":1}
I20260620 19:05:29.672065 8936 tablet_service.cc:1467] Processing CreateTablet for tablet fb45e2200e504be58f61d2b2f4930bcd (DEFAULT_TABLE table=test-table [id=88312e0e18da4512814e06ed8bea89b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:29.672341 8936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb45e2200e504be58f61d2b2f4930bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:29.674084 8662 tablet_service.cc:1467] Processing CreateTablet for tablet fb45e2200e504be58f61d2b2f4930bcd (DEFAULT_TABLE table=test-table [id=88312e0e18da4512814e06ed8bea89b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:29.674367 8662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb45e2200e504be58f61d2b2f4930bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:29.674710 9092 tablet_bootstrap.cc:492] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96: Bootstrap starting.
I20260620 19:05:29.675711 9092 tablet_bootstrap.cc:654] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:29.676788 9094 tablet_bootstrap.cc:492] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc: Bootstrap starting.
I20260620 19:05:29.677000 8799 tablet_service.cc:1467] Processing CreateTablet for tablet fb45e2200e504be58f61d2b2f4930bcd (DEFAULT_TABLE table=test-table [id=88312e0e18da4512814e06ed8bea89b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:29.677271 8799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb45e2200e504be58f61d2b2f4930bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:29.677748 9094 tablet_bootstrap.cc:654] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:29.677953 9092 tablet_bootstrap.cc:492] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96: No bootstrap required, opened a new log
I20260620 19:05:29.678047 9092 ts_tablet_manager.cc:1397] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260620 19:05:29.678551 9092 raft_consensus.cc:348] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } }
I20260620 19:05:29.678704 9092 raft_consensus.cc:374] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:29.678756 9092 raft_consensus.cc:729] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6820357b6174d018dad094c0af49d96, State: Initialized, Role: FOLLOWER
I20260620 19:05:29.678903 9092 consensus_queue.cc:260] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } }
I20260620 19:05:29.679302 9092 ts_tablet_manager.cc:1428] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:05:29.680263 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.679741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.131:41701)
0620 19:05:29.680090 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.680162 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":103,"threads_started":1}
I20260620 19:05:29.680563 9094 tablet_bootstrap.cc:492] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc: No bootstrap required, opened a new log
I20260620 19:05:29.680662 9094 ts_tablet_manager.cc:1397] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 19:05:29.681034 9094 raft_consensus.cc:348] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } }
I20260620 19:05:29.681144 9094 raft_consensus.cc:374] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:29.681187 9094 raft_consensus.cc:729] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b757ebb93849a9836c58f9969cccbc, State: Initialized, Role: FOLLOWER
I20260620 19:05:29.681313 9094 consensus_queue.cc:260] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } }
I20260620 19:05:29.681674 9094 ts_tablet_manager.cc:1428] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:05:29.682365 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.681935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.129:56927)
0620 19:05:29.682168 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.682230 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":78,"threads_started":1}
I20260620 19:05:29.683172 8728 heartbeater.cc:503] Master 127.1.154.188:43659 was elected leader, sending a full tablet report...
I20260620 19:05:29.683934 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.682828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.129:46863)
0620 19:05:29.683790 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.683848 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":892,"thread_start_us":91,"threads_started":1}
I20260620 19:05:29.684185 9002 heartbeater.cc:503] Master 127.1.154.188:43659 was elected leader, sending a full tablet report...
I20260620 19:05:29.684849 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.682471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.131:47891)
0620 19:05:29.684707 (+ 2236us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.684765 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2143}
I20260620 19:05:29.685200 9097 tablet_bootstrap.cc:492] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368: Bootstrap starting.
I20260620 19:05:29.686661 9097 tablet_bootstrap.cc:654] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:29.687736 9097 tablet_bootstrap.cc:492] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368: No bootstrap required, opened a new log
I20260620 19:05:29.687810 9097 ts_tablet_manager.cc:1397] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:29.688153 9097 raft_consensus.cc:348] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } }
I20260620 19:05:29.688244 9097 raft_consensus.cc:374] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:29.688265 9097 raft_consensus.cc:729] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9095510e212429c884037c06eb9b368, State: Initialized, Role: FOLLOWER
I20260620 19:05:29.688417 9097 consensus_queue.cc:260] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } }
I20260620 19:05:29.689496 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.689127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.130:46989)
0620 19:05:29.689347 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.689404 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":77,"threads_started":1}
I20260620 19:05:29.693567 9097 ts_tablet_manager.cc:1428] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260620 19:05:29.693790 8865 heartbeater.cc:503] Master 127.1.154.188:43659 was elected leader, sending a full tablet report...
I20260620 19:05:29.695201 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.694840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.130:46505)
0620 19:05:29.695038 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.695092 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260620 19:05:29.783196 9099 raft_consensus.cc:482] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:29.783352 9099 raft_consensus.cc:504] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } }
I20260620 19:05:29.784694 9099 leader_election.cc:288] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6820357b6174d018dad094c0af49d96 (127.1.154.131:43351), d9095510e212429c884037c06eb9b368 (127.1.154.130:41521)
I20260620 19:05:29.789144 9096 raft_consensus.cc:482] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:29.789265 9096 raft_consensus.cc:504] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } }
I20260620 19:05:29.789366 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.785042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:57075 (local address 127.1.154.131:43351)
0620 19:05:29.785199 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:29.785206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:29.785255 (+ 49us) server_negotiation.cc:407] Connection header received
0620 19:05:29.785520 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:29.785523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:29.785561 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:29.785686 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:29.786127 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.786866 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.788389 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.788811 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.788957 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:29.789010 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:29.789171 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 19:05:29.789241 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 19:05:29.789353 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.785253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:41521 (local address 127.1.154.129:40965)
0620 19:05:29.785330 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.785342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:29.785415 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:29.785822 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:29.785824 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:29.785837 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:29.785945 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:29.785949 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.786992 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.786995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:29.788225 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:29.788234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.788912 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.788916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:29.789005 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:29.789035 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 19:05:29.789119 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:05:29.789198 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":41}
I20260620 19:05:29.789402 9101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.784972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:43351 (local address 127.1.154.129:57075)
0620 19:05:29.785137 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.785154 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:29.785338 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:29.785702 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:29.785706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:29.785725 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:29.785933 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:29.785944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.787009 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.787012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:29.788249 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:29.788258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.788941 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.788944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:29.789053 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:29.789070 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 19:05:29.789162 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 19:05:29.789239 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":47}
I20260620 19:05:29.789700 9096 leader_election.cc:288] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70b757ebb93849a9836c58f9969cccbc (127.1.154.129:45115), d9095510e212429c884037c06eb9b368 (127.1.154.130:41521)
I20260620 19:05:29.789750 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.785381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:40965 (local address 127.1.154.130:41521)
0620 19:05:29.785536 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:29.785540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:29.785555 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:05:29.785650 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:29.785655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:29.785702 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:29.785784 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:29.786061 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.786859 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.788352 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.788778 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.788893 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:29.788952 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:29.789501 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 19:05:29.789617 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:05:29.790015 8682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb45e2200e504be58f61d2b2f4930bcd" candidate_uuid: "d6820357b6174d018dad094c0af49d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70b757ebb93849a9836c58f9969cccbc" is_pre_election: true
I20260620 19:05:29.790081 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb45e2200e504be58f61d2b2f4930bcd" candidate_uuid: "d6820357b6174d018dad094c0af49d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9095510e212429c884037c06eb9b368" is_pre_election: true
I20260620 19:05:29.790073 8956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb45e2200e504be58f61d2b2f4930bcd" candidate_uuid: "70b757ebb93849a9836c58f9969cccbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6820357b6174d018dad094c0af49d96" is_pre_election: true
I20260620 19:05:29.790136 8682 raft_consensus.cc:2454] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6820357b6174d018dad094c0af49d96 in term 0.
I20260620 19:05:29.790189 8819 raft_consensus.cc:2454] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6820357b6174d018dad094c0af49d96 in term 0.
I20260620 19:05:29.790242 8956 raft_consensus.cc:2454] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70b757ebb93849a9836c58f9969cccbc in term 0.
I20260620 19:05:29.790474 8890 leader_election.cc:302] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70b757ebb93849a9836c58f9969cccbc, d6820357b6174d018dad094c0af49d96; no voters:
I20260620 19:05:29.790534 8818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb45e2200e504be58f61d2b2f4930bcd" candidate_uuid: "70b757ebb93849a9836c58f9969cccbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9095510e212429c884037c06eb9b368" is_pre_election: true
I20260620 19:05:29.790628 9096 raft_consensus.cc:2781] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:29.790678 9096 raft_consensus.cc:482] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:29.790653 8818 raft_consensus.cc:2454] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70b757ebb93849a9836c58f9969cccbc in term 0.
I20260620 19:05:29.790711 9096 raft_consensus.cc:3037] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:29.790894 8618 leader_election.cc:302] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70b757ebb93849a9836c58f9969cccbc, d6820357b6174d018dad094c0af49d96; no voters:
I20260620 19:05:29.791347 9099 raft_consensus.cc:2781] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:29.791399 9099 raft_consensus.cc:482] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:29.791425 9099 raft_consensus.cc:3037] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:29.791836 9096 raft_consensus.cc:504] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } }
I20260620 19:05:29.792177 9096 leader_election.cc:288] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [CANDIDATE]: Term 1 election: Requested vote from peers 70b757ebb93849a9836c58f9969cccbc (127.1.154.129:45115), d9095510e212429c884037c06eb9b368 (127.1.154.130:41521)
I20260620 19:05:29.792475 8682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb45e2200e504be58f61d2b2f4930bcd" candidate_uuid: "d6820357b6174d018dad094c0af49d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70b757ebb93849a9836c58f9969cccbc"
I20260620 19:05:29.792505 8818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb45e2200e504be58f61d2b2f4930bcd" candidate_uuid: "d6820357b6174d018dad094c0af49d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9095510e212429c884037c06eb9b368"
I20260620 19:05:29.792619 8818 raft_consensus.cc:3037] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:29.792613 9099 raft_consensus.cc:504] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } }
I20260620 19:05:29.792837 8682 raft_consensus.cc:2379] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d6820357b6174d018dad094c0af49d96 in current term 1: Already voted for candidate 70b757ebb93849a9836c58f9969cccbc in this term.
I20260620 19:05:29.793210 9099 leader_election.cc:288] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [CANDIDATE]: Term 1 election: Requested vote from peers d6820357b6174d018dad094c0af49d96 (127.1.154.131:43351), d9095510e212429c884037c06eb9b368 (127.1.154.130:41521)
I20260620 19:05:29.793459 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb45e2200e504be58f61d2b2f4930bcd" candidate_uuid: "70b757ebb93849a9836c58f9969cccbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9095510e212429c884037c06eb9b368"
I20260620 19:05:29.793886 8818 raft_consensus.cc:2454] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6820357b6174d018dad094c0af49d96 in term 1.
I20260620 19:05:29.793952 8956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb45e2200e504be58f61d2b2f4930bcd" candidate_uuid: "70b757ebb93849a9836c58f9969cccbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6820357b6174d018dad094c0af49d96"
I20260620 19:05:29.794148 8956 raft_consensus.cc:2379] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 70b757ebb93849a9836c58f9969cccbc in current term 1: Already voted for candidate d6820357b6174d018dad094c0af49d96 in this term.
I20260620 19:05:29.794203 8892 leader_election.cc:302] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d6820357b6174d018dad094c0af49d96, d9095510e212429c884037c06eb9b368; no voters: 70b757ebb93849a9836c58f9969cccbc
I20260620 19:05:29.794364 9096 raft_consensus.cc:2781] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:29.794524 8618 leader_election.cc:302] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70b757ebb93849a9836c58f9969cccbc; no voters: d6820357b6174d018dad094c0af49d96, d9095510e212429c884037c06eb9b368
I20260620 19:05:29.794621 9096 raft_consensus.cc:686] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [term 1 LEADER]: Becoming Leader. State: Replica: d6820357b6174d018dad094c0af49d96, State: Running, Role: LEADER
I20260620 19:05:29.794723 9099 raft_consensus.cc:2726] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 19:05:29.794786 9096 consensus_queue.cc:237] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } }
I20260620 19:05:29.795428 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.795197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.131:55979)
0620 19:05:29.795330 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.795372 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 19:05:29.797283 8546 catalog_manager.cc:5495] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6820357b6174d018dad094c0af49d96 (127.1.154.131). New cstate: current_term: 1 leader_uuid: "d6820357b6174d018dad094c0af49d96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6820357b6174d018dad094c0af49d96" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 43351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 } health_report { overall_health: UNKNOWN } } }
I20260620 19:05:29.828333 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.824354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.1.154.131:43351)
0620 19:05:29.824501 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:29.824508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:29.824524 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:05:29.824733 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:29.824736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:29.824788 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:29.824893 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:29.825274 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.826017 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.827160 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:29.827464 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:29.827550 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:29.827851 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:29.827925 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:29.827980 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:29.828091 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 19:05:29.828135 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:05:29.831341 8818 raft_consensus.cc:1261] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368 [term 1 FOLLOWER]: Refusing update from remote peer d6820357b6174d018dad094c0af49d96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:29.831347 8682 raft_consensus.cc:1261] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [term 1 FOLLOWER]: Refusing update from remote peer d6820357b6174d018dad094c0af49d96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:29.832080 9105 consensus_queue.cc:1035] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9095510e212429c884037c06eb9b368" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 41521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:29.832381 9096 consensus_queue.cc:1035] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70b757ebb93849a9836c58f9969cccbc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:29.834438 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.834171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.130:44961)
0620 19:05:29.834294 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.834356 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 19:05:29.835076 9101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.834834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.129:55739)
0620 19:05:29.834976 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.835016 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 19:05:29.837483 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.837089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.131:40513)
0620 19:05:29.837365 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.837416 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260620 19:05:29.838987 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.838588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.131:35839)
0620 19:05:29.838871 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.838918 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260620 19:05:29.840930 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.840677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.130:46931)
0620 19:05:29.840811 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.840861 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 19:05:29.841526 8544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:29.841696 8544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:29.842900 9101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.842553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:42505 (local address 127.1.154.129:57925)
0620 19:05:29.842781 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.842830 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260620 19:05:29.846903 8544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c97bcc01ac84848bd562c2263093c6a: Sending DeleteTablet for 3 replicas of tablet fb45e2200e504be58f61d2b2f4930bcd
I20260620 19:05:29.847514 8936 tablet_service.cc:1514] Processing DeleteTablet for tablet fb45e2200e504be58f61d2b2f4930bcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:29 UTC) from {username='oryx', principal='oryx/127.1.154.188@KRBTEST.COM'} at 127.0.0.1:53874
I20260620 19:05:29.847630 8799 tablet_service.cc:1514] Processing DeleteTablet for tablet fb45e2200e504be58f61d2b2f4930bcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:29 UTC) from {username='oryx', principal='oryx/127.1.154.188@KRBTEST.COM'} at 127.0.0.1:51024
I20260620 19:05:29.848042 9118 tablet_replica.cc:330] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368: stopping tablet replica
I20260620 19:05:29.848186 9118 raft_consensus.cc:2229] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:29.848342 9118 raft_consensus.cc:2258] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:29.848451 8662 tablet_service.cc:1514] Processing DeleteTablet for tablet fb45e2200e504be58f61d2b2f4930bcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:29 UTC) from {username='oryx', principal='oryx/127.1.154.188@KRBTEST.COM'} at 127.0.0.1:33620
I20260620 19:05:29.848546 8661 tablet_service.cc:1514] Processing DeleteTablet for tablet fb45e2200e504be58f61d2b2f4930bcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:29 UTC) from {username='oryx', principal='oryx/127.1.154.188@KRBTEST.COM'} at 127.0.0.1:33620
I20260620 19:05:29.848800 9119 tablet_replica.cc:330] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc: stopping tablet replica
I20260620 19:05:29.848906 9119 raft_consensus.cc:2229] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:29.848943 9117 tablet_replica.cc:330] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96: stopping tablet replica
I20260620 19:05:29.849031 9119 raft_consensus.cc:2258] T fb45e2200e504be58f61d2b2f4930bcd P 70b757ebb93849a9836c58f9969cccbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:29.849068 9117 raft_consensus.cc:2229] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:05:29.849289 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 8509
I20260620 19:05:29.849484 9118 ts_tablet_manager.cc:1905] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:29.849344 9117 raft_consensus.cc:2258] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:29.850756 9117 ts_tablet_manager.cc:1905] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:29.852082 9118 ts_tablet_manager.cc:1918] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:29.852161 9118 log.cc:1196] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/fb45e2200e504be58f61d2b2f4930bcd
I20260620 19:05:29.852454 9118 ts_tablet_manager.cc:1939] T fb45e2200e504be58f61d2b2f4930bcd P d9095510e212429c884037c06eb9b368: Deleting consensus metadata
I20260620 19:05:29.853214 9117 ts_tablet_manager.cc:1918] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:29.853286 9117 log.cc:1196] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/fb45e2200e504be58f61d2b2f4930bcd
I20260620 19:05:29.853505 8536 catalog_manager.cc:4841] TS d9095510e212429c884037c06eb9b368 (127.1.154.130:41521): tablet fb45e2200e504be58f61d2b2f4930bcd (table test-table [id=88312e0e18da4512814e06ed8bea89b7]) successfully deleted
I20260620 19:05:29.853574 9117 ts_tablet_manager.cc:1939] T fb45e2200e504be58f61d2b2f4930bcd P d6820357b6174d018dad094c0af49d96: Deleting consensus metadata
I20260620 19:05:29.854238 8536 catalog_manager.cc:4841] TS d6820357b6174d018dad094c0af49d96 (127.1.154.131:43351): tablet fb45e2200e504be58f61d2b2f4930bcd (table test-table [id=88312e0e18da4512814e06ed8bea89b7]) successfully deleted
I20260620 19:05:29.856768 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 8736
I20260620 19:05:29.857075 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:29.856798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:45115 (local address 127.0.0.1:33632)
0620 19:05:29.856928 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:29.856985 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 19:05:29.857568 8534 catalog_manager.cc:4584] TS 70b757ebb93849a9836c58f9969cccbc (127.1.154.129:45115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb45e2200e504be58f61d2b2f4930bcd: Network error: Client connection negotiation failed: client connection to 127.1.154.129:45115: connect: Connection refused (error 111)
I20260620 19:05:29.862962 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 8872
I20260620 19:05:29.869508 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 8228
I20260620 19:05:29.875403 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 8336
2026-06-20T19:05:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5701 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:05:29 dist-test-slave-6wrf krb5kdc[9122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:05:29 dist-test-slave-6wrf krb5kdc[9122](info): set up 2 sockets
Jun 20 19:05:29 dist-test-slave-6wrf krb5kdc[9122](info): commencing operation
krb5kdc: starting...
W20260620 19:05:30.958328 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:05:31 dist-test-slave-6wrf krb5kdc[9122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:05:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.190.keytab.
Entry for principal oryx/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.190.keytab.
I20260620 19:05:31.096930 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:41221
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.190.keytab
--principal=oryx/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:36151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:41221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:31.208421 9135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:31.208709 9135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:31.208750 9135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:31.213017 9135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:31.213094 9135 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:05:31.213112 9135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:31.213131 9135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:31.213147 9135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:31.213200 9135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:31.219846 9135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:36151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:41221
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.190.keytab
--principal=oryx/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:41221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:31.221487 9135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:31.223115 9135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:05:31.231886 9135 server_base.cc:1034] running on GCE node
W20260620 19:05:31.231838 9140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:31.231824 9143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:31.231842 9141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:31.232542 9135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:31.233645 9135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:31.234874 9135 hybrid_clock.cc:648] HybridClock initialized: now 1781982331234848 us; error 42 us; skew 500 ppm
Jun 20 19:05:31 dist-test-slave-6wrf krb5kdc[9122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982331, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:31.238059 9135 init.cc:377] Logged in from keytab as oryx/127.1.154.190@KRBTEST.COM (short username oryx)
I20260620 19:05:31.239538 9135 webserver.cc:460] Webserver started at http://127.1.154.190:41227/ using document root <none> and password file <none>
I20260620 19:05:31.240267 9135 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:31.240365 9135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:31.240640 9135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:31.243459 9135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "a7a9f01fb7c9447597e059f0b26c14f0"
format_stamp: "Formatted at 2026-06-20 19:05:31 on dist-test-slave-6wrf"
server_key: "afbbb0d27fec73f6e3afae7b49510488"
server_key_iv: "221ad943b7a784b6f3ad952c58637e2c"
server_key_version: "encryptionkey@0"
I20260620 19:05:31.244231 9135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "a7a9f01fb7c9447597e059f0b26c14f0"
format_stamp: "Formatted at 2026-06-20 19:05:31 on dist-test-slave-6wrf"
server_key: "afbbb0d27fec73f6e3afae7b49510488"
server_key_iv: "221ad943b7a784b6f3ad952c58637e2c"
server_key_version: "encryptionkey@0"
I20260620 19:05:31.248744 9135 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:05:31.251264 9150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:31.252827 9135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:05:31.253005 9135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "a7a9f01fb7c9447597e059f0b26c14f0"
format_stamp: "Formatted at 2026-06-20 19:05:31 on dist-test-slave-6wrf"
server_key: "afbbb0d27fec73f6e3afae7b49510488"
server_key_iv: "221ad943b7a784b6f3ad952c58637e2c"
server_key_version: "encryptionkey@0"
I20260620 19:05:31.253129 9135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:31.282261 9135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:31.286604 9135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:31.286864 9135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:31.297809 9135 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:41221
I20260620 19:05:31.297912 9212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:41221 every 8 connection(s)
I20260620 19:05:31.299155 9135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:31.303653 9213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:31.304937 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 9135
I20260620 19:05:31.305140 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:31.305717 1642 external_mini_cluster.cc:1428] Setting key 85919af855c659dcc9858451637b2ea2
I20260620 19:05:31.310441 9213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0: Bootstrap starting.
I20260620 19:05:31.312968 9213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:31.313755 9213 log.cc:824] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:31.316231 9213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0: No bootstrap required, opened a new log
Jun 20 19:05:31 dist-test-slave-6wrf krb5kdc[9122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
I20260620 19:05:31.320994 9213 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a9f01fb7c9447597e059f0b26c14f0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 41221 } }
I20260620 19:05:31.321322 9213 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:31.321373 9213 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a9f01fb7c9447597e059f0b26c14f0, State: Initialized, Role: FOLLOWER
I20260620 19:05:31.321888 9213 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a9f01fb7c9447597e059f0b26c14f0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 41221 } }
I20260620 19:05:31.322046 9213 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:31.322111 9213 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:31.322201 9213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:31.323271 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:31.307274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.1.154.190:41221)
0620 19:05:31.307700 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:31.307715 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:31.307751 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:05:31.308380 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:31.308396 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:31.308710 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:31.308976 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:31.310139 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.311448 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.312717 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.313454 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.313826 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:31.317166 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:31.317174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:31.317190 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:31.317222 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:31.319512 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:31.320175 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:31.320181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:31.320188 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:31.320335 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:31.320760 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:31.320763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:31.320765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:31.321452 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:31.321690 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:31.322008 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 19:05:31.322316 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":143,"threads_started":1}
I20260620 19:05:31.323346 9213 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a9f01fb7c9447597e059f0b26c14f0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 41221 } }
I20260620 19:05:31.323669 9213 leader_election.cc:302] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7a9f01fb7c9447597e059f0b26c14f0; no voters:
I20260620 19:05:31.323982 9213 leader_election.cc:288] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:31.324131 9218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:31.324361 9218 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [term 1 LEADER]: Becoming Leader. State: Replica: a7a9f01fb7c9447597e059f0b26c14f0, State: Running, Role: LEADER
I20260620 19:05:31.324656 9218 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a9f01fb7c9447597e059f0b26c14f0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 41221 } }
I20260620 19:05:31.325263 9213 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:31.326436 9219 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7a9f01fb7c9447597e059f0b26c14f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a9f01fb7c9447597e059f0b26c14f0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 41221 } } }
I20260620 19:05:31.326561 9219 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:31.326792 9220 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7a9f01fb7c9447597e059f0b26c14f0. Latest consensus state: current_term: 1 leader_uuid: "a7a9f01fb7c9447597e059f0b26c14f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a9f01fb7c9447597e059f0b26c14f0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 41221 } } }
I20260620 19:05:31.326879 9220 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:31.327009 9227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:31.329536 9227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:31.334673 9227 catalog_manager.cc:1347] Generated new cluster ID: 5c98452583e940419a1697bd55e6f440
I20260620 19:05:31.334735 9227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:31.365384 9227 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:31.366420 9227 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:31.372617 9227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0: Generated new TSK 0
I20260620 19:05:31.373291 9227 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.129.keytab.
Entry for principal oryx/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.129.keytab.
I20260620 19:05:31.410209 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:41221
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.129.keytab
--principal=oryx/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:36151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:31.535609 9239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:31.535940 9239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:31.535991 9239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:31.539995 9239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:31.540057 9239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:31.540097 9239 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:05:31.540130 9239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:05:31.545496 9239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:36151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.129.keytab
--principal=oryx/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.154.190:41221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:31.546662 9239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:31.547628 9239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:31.557981 9247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:31.558073 9245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:31.558010 9244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:31.558300 9239 server_base.cc:1034] running on GCE node
I20260620 19:05:31.558773 9239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:31.559494 9239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:31.560667 9239 hybrid_clock.cc:648] HybridClock initialized: now 1781982331560629 us; error 43 us; skew 500 ppm
Jun 20 19:05:31 dist-test-slave-6wrf krb5kdc[9122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982331, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:31.563539 9239 init.cc:377] Logged in from keytab as oryx/127.1.154.129@KRBTEST.COM (short username oryx)
I20260620 19:05:31.564462 9239 webserver.cc:460] Webserver started at http://127.1.154.129:44349/ using document root <none> and password file <none>
I20260620 19:05:31.565131 9239 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:31.565234 9239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:31.565467 9239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:31.567574 9239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "a71340573c8c4984a53d4119e70706f4"
format_stamp: "Formatted at 2026-06-20 19:05:31 on dist-test-slave-6wrf"
server_key: "3fa88f31fd09eda4c49f10f4784adc33"
server_key_iv: "052b2df013fa24c09f5f12b9f14f6da2"
server_key_version: "encryptionkey@0"
I20260620 19:05:31.568181 9239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "a71340573c8c4984a53d4119e70706f4"
format_stamp: "Formatted at 2026-06-20 19:05:31 on dist-test-slave-6wrf"
server_key: "3fa88f31fd09eda4c49f10f4784adc33"
server_key_iv: "052b2df013fa24c09f5f12b9f14f6da2"
server_key_version: "encryptionkey@0"
I20260620 19:05:31.572557 9239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:05:31.575441 9254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:31.576684 9239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:05:31.576828 9239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "a71340573c8c4984a53d4119e70706f4"
format_stamp: "Formatted at 2026-06-20 19:05:31 on dist-test-slave-6wrf"
server_key: "3fa88f31fd09eda4c49f10f4784adc33"
server_key_iv: "052b2df013fa24c09f5f12b9f14f6da2"
server_key_version: "encryptionkey@0"
I20260620 19:05:31.576979 9239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:31.588847 9239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:31.592348 9239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:31.592583 9239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:31.594184 9239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:31.594255 9239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:31.594318 9239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:31.594365 9239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:05:31 dist-test-slave-6wrf krb5kdc[9122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982331, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
I20260620 19:05:31.605448 9239 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:46449
I20260620 19:05:31.605680 9369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:46449 every 8 connection(s)
I20260620 19:05:31.606720 9239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:05:31.607213 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 9239
I20260620 19:05:31.607336 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:05:31.607681 1642 external_mini_cluster.cc:1428] Setting key 1582a51bd723c78eeeb53ade5260f619
I20260620 19:05:31.609930 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:31.594942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:40085 (local address 127.1.154.190:41221)
0620 19:05:31.595139 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:31.595145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:31.595987 (+ 842us) server_negotiation.cc:407] Connection header received
0620 19:05:31.597036 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:31.597042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:31.597104 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:31.597166 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:31.598567 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.599501 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.600802 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.601165 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.601286 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:31.604719 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:31.604726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:31.604732 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:31.604769 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:31.606625 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:31.607772 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:31.607776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:31.607778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:31.607909 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:31.608410 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:31.608413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:31.608415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:31.608699 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:31.608795 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:31.609553 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 19:05:31.609727 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 19:05:31.610678 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:31.595380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:41221 (local address 127.1.154.129:40085)
0620 19:05:31.595825 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:31.595873 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:31.596796 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:31.597520 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:31.597526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:31.597898 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:31.598372 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:31.598384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.599642 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.599648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:31.600679 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:31.600688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.601329 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.601333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:31.601515 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:31.602201 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:31.602224 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:31.604483 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:31.606893 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:31.606905 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:31.606936 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:31.607441 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:31.608062 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:31.608067 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:31.608070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:31.608299 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:31.608826 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:31.608833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:31.609227 (+ 394us) client_negotiation.cc:769] Sending connection context
0620 19:05:31.609449 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 19:05:31.609787 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":124,"threads_started":1}
I20260620 19:05:31.612567 9370 heartbeater.cc:347] Connected to a master server at 127.1.154.190:41221
I20260620 19:05:31.612739 9370 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:31.613343 9370 heartbeater.cc:511] Master 127.1.154.190:41221 requested a full tablet report, sending...
I20260620 19:05:31.615703 9166 ts_manager.cc:194] Registered new tserver with Master: a71340573c8c4984a53d4119e70706f4 (127.1.154.129:46449)
I20260620 19:05:31.617203 9166 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:40085
WARNING: no policy specified for oryx/127.1.154.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.154.130@KRBTEST.COM" created.
I20260620 19:05:31.625958 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:31.615679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:41221 (local address 127.1.154.129:33483)
0620 19:05:31.617869 (+ 2190us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:31.617889 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:31.618082 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:31.618403 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:31.618409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:31.618648 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:31.618812 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:31.618819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.619773 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.619777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:31.620438 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:31.620447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.621052 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.621054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:31.621127 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:31.621776 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:31.621788 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:31.622121 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:31.623923 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:31.623937 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:31.623942 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:31.624198 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:31.624484 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:31.624490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:31.624495 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:31.624596 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:31.625130 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:31.625136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:31.625253 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 19:05:31.625382 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 19:05:31.625552 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2058,"thread_start_us":132,"threads_started":1}
I20260620 19:05:31.626156 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:31.615774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:33483 (local address 127.1.154.190:41221)
0620 19:05:31.615926 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:31.615931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:31.617992 (+ 2061us) server_negotiation.cc:407] Connection header received
0620 19:05:31.618217 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:31.618221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:31.618275 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:31.618362 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:31.618931 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.619625 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.620686 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.620911 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.621010 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:31.622276 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:31.622281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:31.622283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:31.622310 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:31.623793 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:31.624296 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:31.624306 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:31.624308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:31.624388 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:31.624773 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:31.624776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:31.624778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:31.625029 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:31.625117 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:31.625507 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 19:05:31.625671 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.130.keytab.
Entry for principal oryx/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.130.keytab.
I20260620 19:05:31.642345 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:41221
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.130.keytab
--principal=oryx/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:36151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:31.765851 9379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:31.766163 9379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:31.766235 9379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:31.771332 9379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:31.771798 9379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:31.771976 9379 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:05:31.772061 9379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:31.780601 9379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:36151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.130.keytab
--principal=oryx/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.154.190:41221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:31.781975 9379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:31.783113 9379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:05:31.793588 9379 server_base.cc:1034] running on GCE node
W20260620 19:05:31.793574 9384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:31.793573 9385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:31.793630 9387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:31.794346 9379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:31.795077 9379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:31.796329 9379 hybrid_clock.cc:648] HybridClock initialized: now 1781982331796263 us; error 60 us; skew 500 ppm
Jun 20 19:05:31 dist-test-slave-6wrf krb5kdc[9122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982331, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:31.799943 9379 init.cc:377] Logged in from keytab as oryx/127.1.154.130@KRBTEST.COM (short username oryx)
I20260620 19:05:31.800875 9379 webserver.cc:460] Webserver started at http://127.1.154.130:43399/ using document root <none> and password file <none>
I20260620 19:05:31.801478 9379 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:31.801544 9379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:31.801812 9379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:31.803920 9379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "ec0c4b27112c4d6695befd0528a61553"
format_stamp: "Formatted at 2026-06-20 19:05:31 on dist-test-slave-6wrf"
server_key: "773f9c605dcc51f236582dd663d4918b"
server_key_iv: "34d8d52b5ea69e57502ef4ad3efc5fe5"
server_key_version: "encryptionkey@0"
I20260620 19:05:31.804520 9379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "ec0c4b27112c4d6695befd0528a61553"
format_stamp: "Formatted at 2026-06-20 19:05:31 on dist-test-slave-6wrf"
server_key: "773f9c605dcc51f236582dd663d4918b"
server_key_iv: "34d8d52b5ea69e57502ef4ad3efc5fe5"
server_key_version: "encryptionkey@0"
I20260620 19:05:31.808885 9379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:05:31.812608 9394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:31.814440 9379 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:05:31.814594 9379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "ec0c4b27112c4d6695befd0528a61553"
format_stamp: "Formatted at 2026-06-20 19:05:31 on dist-test-slave-6wrf"
server_key: "773f9c605dcc51f236582dd663d4918b"
server_key_iv: "34d8d52b5ea69e57502ef4ad3efc5fe5"
server_key_version: "encryptionkey@0"
I20260620 19:05:31.814785 9379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:31.846526 9379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:31.849756 9379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:31.849915 9379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:31.851315 9379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:31.851403 9379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:31.851466 9379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:31.851516 9379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:05:31 dist-test-slave-6wrf krb5kdc[9122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982331, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
I20260620 19:05:31.862394 9379 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:40951
I20260620 19:05:31.862452 9509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:40951 every 8 connection(s)
I20260620 19:05:31.863399 9379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:31.866165 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:31.852355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:59147 (local address 127.1.154.190:41221)
0620 19:05:31.852587 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:31.852591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:31.853473 (+ 882us) server_negotiation.cc:407] Connection header received
0620 19:05:31.854559 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:31.854564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:31.854655 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:31.854719 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:31.856461 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.857351 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.858882 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.859140 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.859255 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:31.862085 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:31.862089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:31.862092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:31.862122 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:31.863567 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:31.864153 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:31.864158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:31.864163 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:31.864246 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:31.864670 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:31.864672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:31.864674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:31.864865 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:31.864947 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:31.865937 (+ 990us) server_negotiation.cc:299] Negotiation successful
0620 19:05:31.866039 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 19:05:31.867434 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:31.852732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:41221 (local address 127.1.154.130:59147)
0620 19:05:31.853260 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:31.853307 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:31.854313 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:31.855193 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:31.855202 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:31.855599 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:31.856295 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:31.856312 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.857500 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.857506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:31.858751 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:31.858763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.859295 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.859298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:31.859473 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:31.860056 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:31.860072 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:31.861894 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:31.863681 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:31.863687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:31.863700 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:31.864034 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:31.864388 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:31.864391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:31.864392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:31.864550 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:31.865090 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:31.865099 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:31.865480 (+ 381us) client_negotiation.cc:769] Sending connection context
0620 19:05:31.865837 (+ 357us) client_negotiation.cc:240] Negotiation successful
0620 19:05:31.866223 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":146,"threads_started":1}
I20260620 19:05:31.868956 9510 heartbeater.cc:347] Connected to a master server at 127.1.154.190:41221
I20260620 19:05:31.869110 9510 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:31.869671 9510 heartbeater.cc:511] Master 127.1.154.190:41221 requested a full tablet report, sending...
I20260620 19:05:31.870879 9166 ts_manager.cc:194] Registered new tserver with Master: ec0c4b27112c4d6695befd0528a61553 (127.1.154.130:40951)
I20260620 19:05:31.871150 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 9379
I20260620 19:05:31.871305 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:05:31.871603 9166 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:59147
I20260620 19:05:31.871660 1642 external_mini_cluster.cc:1428] Setting key 5d15b64a77e67bd81c7207fc49febba1
I20260620 19:05:31.882682 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:31.872886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:35887 (local address 127.1.154.190:41221)
0620 19:05:31.873167 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:31.873173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:31.873271 (+ 98us) server_negotiation.cc:407] Connection header received
0620 19:05:31.873395 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:31.873398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:31.873444 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:31.873534 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:31.874170 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.874700 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.875865 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.876355 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.876545 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:31.878164 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:31.878168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:31.878170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:31.878193 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:31.880603 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:31.881265 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:31.881269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:31.881270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:31.881405 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:31.881762 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:31.881764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:31.881766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:31.882028 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:31.882109 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:31.882464 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 19:05:31.882565 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 19:05:31.882794 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:31.872731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:41221 (local address 127.1.154.130:35887)
0620 19:05:31.873169 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:31.873185 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:31.873301 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:31.873580 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:31.873583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:31.873847 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:31.874007 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:31.874018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.874830 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.874837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:31.875686 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:31.875696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:31.876542 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:31.876545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:31.876626 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:31.877322 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:31.877344 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:31.877982 (+ 638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:31.880761 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:31.880765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:31.880768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:31.881103 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:31.881520 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:31.881524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:31.881528 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:31.881666 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:31.882131 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:31.882135 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:31.882244 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 19:05:31.882369 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 19:05:31.882548 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for oryx/127.1.154.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.154.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.131.keytab.
Entry for principal oryx/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.131.keytab.
I20260620 19:05:31.907217 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:41221
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.131.keytab
--principal=oryx/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:36151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:32.038487 9519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:32.038841 9519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:32.038928 9519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:32.043891 9519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:32.043998 9519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:32.044072 9519 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:05:32.044150 9519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:32.049818 9519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:36151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.131.keytab
--principal=oryx/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.154.190:41221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:32.050948 9519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:32.052016 9519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:32.062001 9525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:32.062017 9524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:32.062026 9527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:32.062228 9519 server_base.cc:1034] running on GCE node
I20260620 19:05:32.062736 9519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:32.063441 9519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:32.064640 9519 hybrid_clock.cc:648] HybridClock initialized: now 1781982332064619 us; error 33 us; skew 500 ppm
Jun 20 19:05:32 dist-test-slave-6wrf krb5kdc[9122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982332, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:32.068228 9519 init.cc:377] Logged in from keytab as oryx/127.1.154.131@KRBTEST.COM (short username oryx)
I20260620 19:05:32.069794 9519 webserver.cc:460] Webserver started at http://127.1.154.131:40683/ using document root <none> and password file <none>
I20260620 19:05:32.070544 9519 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:32.070636 9519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:32.070952 9519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:32.073081 9519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "729762e4e989441cbd1c684bf6eed114"
format_stamp: "Formatted at 2026-06-20 19:05:32 on dist-test-slave-6wrf"
server_key: "64dc620f838e4b370ec63b8782a7e3dc"
server_key_iv: "350fde46535cc159f575104cc1907e5a"
server_key_version: "encryptionkey@0"
I20260620 19:05:32.073725 9519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "729762e4e989441cbd1c684bf6eed114"
format_stamp: "Formatted at 2026-06-20 19:05:32 on dist-test-slave-6wrf"
server_key: "64dc620f838e4b370ec63b8782a7e3dc"
server_key_iv: "350fde46535cc159f575104cc1907e5a"
server_key_version: "encryptionkey@0"
I20260620 19:05:32.078125 9519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:05:32.081382 9534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:32.082748 9519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:32.082890 9519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "729762e4e989441cbd1c684bf6eed114"
format_stamp: "Formatted at 2026-06-20 19:05:32 on dist-test-slave-6wrf"
server_key: "64dc620f838e4b370ec63b8782a7e3dc"
server_key_iv: "350fde46535cc159f575104cc1907e5a"
server_key_version: "encryptionkey@0"
I20260620 19:05:32.083014 9519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:32.099140 9519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:32.103570 9519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:32.103812 9519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:32.105329 9519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:32.105401 9519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:32.105465 9519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:32.105511 9519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:05:32 dist-test-slave-6wrf krb5kdc[9122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982332, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
I20260620 19:05:32.119163 9519 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:35867
I20260620 19:05:32.119254 9649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:35867 every 8 connection(s)
I20260620 19:05:32.120908 9519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:32.122845 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.106254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:38723 (local address 127.1.154.190:41221)
0620 19:05:32.106442 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.106447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.107402 (+ 955us) server_negotiation.cc:407] Connection header received
0620 19:05:32.108585 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.108591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.108659 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.108788 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:32.110819 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.111886 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.114211 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.114461 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.114582 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.117500 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:32.117507 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:32.117511 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:32.117546 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:32.119740 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:32.120400 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:32.120407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:32.120412 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:32.120654 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:32.121219 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:32.121225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:32.121229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:32.121496 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:32.121661 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.122522 (+ 861us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.122671 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 19:05:32.123768 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.106744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:41221 (local address 127.1.154.131:38723)
0620 19:05:32.107226 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.107274 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.108189 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.109321 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.109332 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.109928 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:32.110612 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.110631 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.112073 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.112076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.114035 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.114056 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.114638 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.114642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.114868 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.115572 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:32.115595 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:32.117281 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:32.119942 (+ 2661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:32.119947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:32.119960 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:32.120253 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:32.120841 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:32.120845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:32.120847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:32.121077 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:32.121686 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:32.121694 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:32.122201 (+ 507us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.122456 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.122818 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":129,"threads_started":1}
I20260620 19:05:32.125200 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 9519
I20260620 19:05:32.125281 9650 heartbeater.cc:347] Connected to a master server at 127.1.154.190:41221
I20260620 19:05:32.125340 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:05:32.125424 9650 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:32.125705 1642 external_mini_cluster.cc:1428] Setting key 4ef64825a9a4611d24ec11ada88dc9f6
I20260620 19:05:32.126058 9650 heartbeater.cc:511] Master 127.1.154.190:41221 requested a full tablet report, sending...
I20260620 19:05:32.127147 9166 ts_manager.cc:194] Registered new tserver with Master: 729762e4e989441cbd1c684bf6eed114 (127.1.154.131:35867)
I20260620 19:05:32.127954 9166 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:38723
I20260620 19:05:32.128006 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:05:32.136620 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.127743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:41221 (local address 127.1.154.131:49025)
0620 19:05:32.128024 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.128040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.128140 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.128376 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.128379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.128594 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:32.128747 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.128754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.129810 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.129813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.130496 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.130504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.130979 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.130983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.131077 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.131616 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:32.131634 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:32.132066 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:32.134102 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:32.134105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:32.134107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:32.134325 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:32.134678 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:32.134683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:32.134687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:32.134803 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:32.135812 (+ 1009us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:32.135818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:32.136041 (+ 223us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.136219 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.136394 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":5,"thread_start_us":115,"threads_started":1}
I20260620 19:05:32.137257 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.127782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:49025 (local address 127.1.154.190:41221)
0620 19:05:32.128038 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.128043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.128096 (+ 53us) server_negotiation.cc:407] Connection header received
0620 19:05:32.128232 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.128235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.128275 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.128361 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:32.128879 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.129654 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.130631 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.130866 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.130977 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.132180 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:32.132184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:32.132187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:32.132209 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:32.133941 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:32.134477 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:32.134482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:32.134487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:32.134577 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:32.135118 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:32.135124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:32.135126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:32.135515 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:32.135707 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.136916 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.137055 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
Jun 20 19:05:32 dist-test-slave-6wrf krb5kdc[9122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:05:32 dist-test-slave-6wrf krb5kdc[9122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:32 dist-test-slave-6wrf krb5kdc[9122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:05:32.159178 9216 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:32.150702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35186 (local address 127.1.154.190:41221)
0620 19:05:32.150855 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.150859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.150872 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:05:32.150936 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.150938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.150977 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.151051 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:32.152309 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.153001 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.154173 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.154430 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.154754 (+ 324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.158994 (+ 4240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35186: BlockingRecv error: recv got EOF from 127.0.0.1:35186 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 19:05:32 dist-test-slave-6wrf krb5kdc[9122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
I20260620 19:05:32.170033 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.160876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35198 (local address 127.1.154.190:41221)
0620 19:05:32.161032 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.161036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.161048 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:05:32.161088 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.161090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.161126 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.161211 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:32.161858 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.162373 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.163139 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.163358 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.163463 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.166557 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:32.166560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:32.166562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:32.166585 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:32.167925 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:32.168702 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:32.168705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:32.168706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:32.168809 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:32.169175 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:32.169178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:32.169179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:32.169373 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:32.169492 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.169758 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.169853 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 19:05:32.174031 9166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:05:32.176956 9166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:32.191412 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.185554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:35867 (local address 127.0.0.1:60926)
0620 19:05:32.186082 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.186095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.186180 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.187185 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.187188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.187209 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:32.187296 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.187302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.188760 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.188765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.189940 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.189950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.190900 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.190904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.191011 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.191049 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.191153 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.191256 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":19,"thread_start_us":119,"threads_started":1}
I20260620 19:05:32.191443 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.186053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60926 (local address 127.1.154.131:35867)
0620 19:05:32.186752 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.186773 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.186790 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:05:32.186860 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.186864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.187025 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.187170 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:32.187468 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.188626 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.190048 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.190781 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.190894 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.191054 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.191193 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.191285 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":106,"threads_started":1}
I20260620 19:05:32.192363 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.185909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:46449 (local address 127.0.0.1:51996)
0620 19:05:32.186372 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.186394 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.186480 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.187201 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.187204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.187220 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:32.187332 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.187337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.188846 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.188849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.190631 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.190637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.191968 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.191973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.192081 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.192105 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.192154 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.192221 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":710,"thread_start_us":81,"threads_started":1}
I20260620 19:05:32.192457 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.185344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.1.154.130:40951)
0620 19:05:32.186026 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.186037 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.186069 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:05:32.186146 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.186156 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.186418 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.186595 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:32.187016 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.187954 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.190532 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.191298 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.191421 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.192150 (+ 729us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.192199 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.192316 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":110,"threads_started":1}
I20260620 19:05:32.192708 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.185186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:40951 (local address 127.0.0.1:50408)
0620 19:05:32.185564 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.185633 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.185763 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.186772 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.186780 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.186803 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:32.186907 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.186912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.188055 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.188059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.190431 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.190439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.191723 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.191726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.191812 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.191831 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.191892 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.192571 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260620 19:05:32.193989 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.186523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.1.154.129:46449)
0620 19:05:32.186752 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.186760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.186783 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:05:32.186854 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.186858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.187068 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.187502 (+ 434us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:32.187615 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.188740 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.190735 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.191814 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.193556 (+ 1742us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.193731 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.193779 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.193865 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260620 19:05:32.194631 9584 tablet_service.cc:1467] Processing CreateTablet for tablet 1baaf235825346b2904b2ebc7df0c5f7 (DEFAULT_TABLE table=test-table [id=4a661c65fe004fe6a8004c384f8f95c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:32.195050 9444 tablet_service.cc:1467] Processing CreateTablet for tablet 1baaf235825346b2904b2ebc7df0c5f7 (DEFAULT_TABLE table=test-table [id=4a661c65fe004fe6a8004c384f8f95c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:32.195672 9584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1baaf235825346b2904b2ebc7df0c5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:32.195959 9444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1baaf235825346b2904b2ebc7df0c5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:32.196285 9304 tablet_service.cc:1467] Processing CreateTablet for tablet 1baaf235825346b2904b2ebc7df0c5f7 (DEFAULT_TABLE table=test-table [id=4a661c65fe004fe6a8004c384f8f95c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:32.197161 9304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1baaf235825346b2904b2ebc7df0c5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:32.204384 9678 tablet_bootstrap.cc:492] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114: Bootstrap starting.
I20260620 19:05:32.205053 9679 tablet_bootstrap.cc:492] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4: Bootstrap starting.
I20260620 19:05:32.205941 9680 tablet_bootstrap.cc:492] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553: Bootstrap starting.
I20260620 19:05:32.206904 9678 tablet_bootstrap.cc:654] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:32.207033 9679 tablet_bootstrap.cc:654] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:32.207867 9679 log.cc:824] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:32.207911 9680 tablet_bootstrap.cc:654] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:32.208209 9678 log.cc:824] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:32.208642 9680 log.cc:824] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:32.209991 9678 tablet_bootstrap.cc:492] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114: No bootstrap required, opened a new log
I20260620 19:05:32.209991 9679 tablet_bootstrap.cc:492] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4: No bootstrap required, opened a new log
I20260620 19:05:32.210187 9679 ts_tablet_manager.cc:1397] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 19:05:32.210187 9678 ts_tablet_manager.cc:1397] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:05:32.210960 9680 tablet_bootstrap.cc:492] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553: No bootstrap required, opened a new log
I20260620 19:05:32.211140 9680 ts_tablet_manager.cc:1397] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:05:32.214308 9680 raft_consensus.cc:348] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.214404 9678 raft_consensus.cc:348] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.214581 9680 raft_consensus.cc:374] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:32.214632 9680 raft_consensus.cc:729] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0c4b27112c4d6695befd0528a61553, State: Initialized, Role: FOLLOWER
I20260620 19:05:32.214636 9678 raft_consensus.cc:374] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:32.214684 9678 raft_consensus.cc:729] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729762e4e989441cbd1c684bf6eed114, State: Initialized, Role: FOLLOWER
I20260620 19:05:32.214670 9679 raft_consensus.cc:348] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.214892 9679 raft_consensus.cc:374] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:32.214936 9679 raft_consensus.cc:729] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71340573c8c4984a53d4119e70706f4, State: Initialized, Role: FOLLOWER
I20260620 19:05:32.215118 9678 consensus_queue.cc:260] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.215162 9680 consensus_queue.cc:260] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.215253 9679 consensus_queue.cc:260] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.215803 9650 heartbeater.cc:503] Master 127.1.154.190:41221 was elected leader, sending a full tablet report...
I20260620 19:05:32.216131 9680 ts_tablet_manager.cc:1428] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:05:32.216146 9678 ts_tablet_manager.cc:1428] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:05:32.216136 9679 ts_tablet_manager.cc:1428] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:05:32.216415 9510 heartbeater.cc:503] Master 127.1.154.190:41221 was elected leader, sending a full tablet report...
I20260620 19:05:32.216537 9370 heartbeater.cc:503] Master 127.1.154.190:41221 was elected leader, sending a full tablet report...
I20260620 19:05:32.275681 9686 raft_consensus.cc:482] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:32.275928 9686 raft_consensus.cc:504] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.277109 9686 leader_election.cc:288] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a71340573c8c4984a53d4119e70706f4 (127.1.154.129:46449), 729762e4e989441cbd1c684bf6eed114 (127.1.154.131:35867)
I20260620 19:05:32.281795 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.277438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:35867 (local address 127.1.154.130:54187)
0620 19:05:32.277575 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.277616 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.277733 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.278107 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.278110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.278187 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:32.278376 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.278393 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.279368 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.279374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.280404 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.280417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.281229 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.281233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.281347 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.281408 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.281494 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.281621 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 19:05:32.281795 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.277530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:54187 (local address 127.1.154.131:35867)
0620 19:05:32.277666 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.277670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.277688 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:05:32.277846 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.277849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.277888 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.277933 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:32.278533 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.279212 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.280607 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.281107 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.281218 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.281260 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.281547 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.281653 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:05:32.282488 9604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1baaf235825346b2904b2ebc7df0c5f7" candidate_uuid: "ec0c4b27112c4d6695befd0528a61553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "729762e4e989441cbd1c684bf6eed114" is_pre_election: true
I20260620 19:05:32.282840 9604 raft_consensus.cc:2454] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec0c4b27112c4d6695befd0528a61553 in term 0.
I20260620 19:05:32.282917 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.277557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:36389 (local address 127.1.154.129:46449)
0620 19:05:32.277692 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.277696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.278629 (+ 933us) server_negotiation.cc:407] Connection header received
0620 19:05:32.278722 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.278725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.278771 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.278862 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:32.279762 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.280322 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.281657 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.282324 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.282426 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.282478 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.282737 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.282807 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:05:32.282943 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.277444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:46449 (local address 127.1.154.130:36389)
0620 19:05:32.278508 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.278526 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.278634 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.278918 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.278921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.278952 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:32.279569 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.279578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.280442 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.280448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.281464 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.281476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.282504 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.282510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.282601 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.282642 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.282722 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.282795 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":560,"thread_start_us":109,"threads_started":1}
I20260620 19:05:32.283356 9395 leader_election.cc:302] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 729762e4e989441cbd1c684bf6eed114, ec0c4b27112c4d6695befd0528a61553; no voters:
I20260620 19:05:32.283465 9324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1baaf235825346b2904b2ebc7df0c5f7" candidate_uuid: "ec0c4b27112c4d6695befd0528a61553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71340573c8c4984a53d4119e70706f4" is_pre_election: true
I20260620 19:05:32.283631 9686 raft_consensus.cc:2781] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:32.283730 9686 raft_consensus.cc:482] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:32.283790 9686 raft_consensus.cc:3037] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:32.283787 9324 raft_consensus.cc:2454] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec0c4b27112c4d6695befd0528a61553 in term 0.
I20260620 19:05:32.284940 9686 raft_consensus.cc:504] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.285274 9686 leader_election.cc:288] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [CANDIDATE]: Term 1 election: Requested vote from peers a71340573c8c4984a53d4119e70706f4 (127.1.154.129:46449), 729762e4e989441cbd1c684bf6eed114 (127.1.154.131:35867)
I20260620 19:05:32.285835 9604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1baaf235825346b2904b2ebc7df0c5f7" candidate_uuid: "ec0c4b27112c4d6695befd0528a61553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "729762e4e989441cbd1c684bf6eed114"
I20260620 19:05:32.285856 9324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1baaf235825346b2904b2ebc7df0c5f7" candidate_uuid: "ec0c4b27112c4d6695befd0528a61553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71340573c8c4984a53d4119e70706f4"
I20260620 19:05:32.286036 9604 raft_consensus.cc:3037] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:32.286058 9324 raft_consensus.cc:3037] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:32.287195 9324 raft_consensus.cc:2454] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec0c4b27112c4d6695befd0528a61553 in term 1.
I20260620 19:05:32.287314 9604 raft_consensus.cc:2454] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec0c4b27112c4d6695befd0528a61553 in term 1.
I20260620 19:05:32.287696 9396 leader_election.cc:302] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a71340573c8c4984a53d4119e70706f4, ec0c4b27112c4d6695befd0528a61553; no voters:
I20260620 19:05:32.287951 9686 raft_consensus.cc:2781] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:32.288354 9686 raft_consensus.cc:686] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [term 1 LEADER]: Becoming Leader. State: Replica: ec0c4b27112c4d6695befd0528a61553, State: Running, Role: LEADER
I20260620 19:05:32.288715 9686 consensus_queue.cc:237] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.291903 9165 catalog_manager.cc:5495] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec0c4b27112c4d6695befd0528a61553 (127.1.154.130). New cstate: current_term: 1 leader_uuid: "ec0c4b27112c4d6695befd0528a61553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } health_report { overall_health: UNKNOWN } } }
I20260620 19:05:32.317090 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.313204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.1.154.130:40951)
0620 19:05:32.313397 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.313402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.313421 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:05:32.313635 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.313642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.313729 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.313843 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:32.314047 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.314668 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.315611 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.315863 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.315968 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.316194 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:32.316675 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:32.316763 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.316844 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.316904 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 19:05:32.324016 9324 raft_consensus.cc:1261] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Refusing update from remote peer ec0c4b27112c4d6695befd0528a61553: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:32.324375 9604 raft_consensus.cc:1261] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Refusing update from remote peer ec0c4b27112c4d6695befd0528a61553: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:32.324687 9686 consensus_queue.cc:1035] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:32.325089 9688 consensus_queue.cc:1035] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:32.332895 9692 mvcc.cc:205] Tried to move back new op lower bound from 7298999633192415232 to 7298999633057353728. Current Snapshot: MvccSnapshot[applied={T|T < 7298999633192415232}]
I20260620 19:05:32.333127 9693 mvcc.cc:205] Tried to move back new op lower bound from 7298999633192415232 to 7298999633057353728. Current Snapshot: MvccSnapshot[applied={T|T < 7298999633192415232}]
I20260620 19:05:32.335469 9691 mvcc.cc:205] Tried to move back new op lower bound from 7298999633192415232 to 7298999633057353728. Current Snapshot: MvccSnapshot[applied={T|T < 7298999633192415232 or (T in {7298999633192415232})}]
I20260620 19:05:32.341290 9165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:32.341480 9165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:32.344065 9165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0: Sending DeleteTablet for 3 replicas of tablet 1baaf235825346b2904b2ebc7df0c5f7
I20260620 19:05:32.344941 9444 tablet_service.cc:1514] Processing DeleteTablet for tablet 1baaf235825346b2904b2ebc7df0c5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:32 UTC) from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:50408
I20260620 19:05:32.344946 9584 tablet_service.cc:1514] Processing DeleteTablet for tablet 1baaf235825346b2904b2ebc7df0c5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:32 UTC) from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:60926
I20260620 19:05:32.345078 9304 tablet_service.cc:1514] Processing DeleteTablet for tablet 1baaf235825346b2904b2ebc7df0c5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:32 UTC) from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:51996
I20260620 19:05:32.345674 9703 tablet_replica.cc:330] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553: stopping tablet replica
I20260620 19:05:32.345762 9702 tablet_replica.cc:330] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114: stopping tablet replica
I20260620 19:05:32.346088 9702 raft_consensus.cc:2229] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:32.346088 9703 raft_consensus.cc:2229] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:05:32.346323 9165 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:05:32.346438 9702 raft_consensus.cc:2258] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:32.346730 9703 raft_consensus.cc:2258] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 19:05:32.346736 9165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:32.346833 9704 tablet_replica.cc:330] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4: stopping tablet replica
I20260620 19:05:32.347116 9704 raft_consensus.cc:2229] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:32.347420 9704 raft_consensus.cc:2258] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:32.349228 9704 ts_tablet_manager.cc:1905] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:32.350622 9703 ts_tablet_manager.cc:1905] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:32.351824 9584 tablet_service.cc:1467] Processing CreateTablet for tablet 377d9fdda7334e48bfe79c05625b48fc (DEFAULT_TABLE table=test-table [id=14e56586c1174edfb942c9ea3c438c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:32.352097 9584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377d9fdda7334e48bfe79c05625b48fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:32.352240 9704 ts_tablet_manager.cc:1918] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:32.352351 9704 log.cc:1196] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/wal/wals/1baaf235825346b2904b2ebc7df0c5f7
I20260620 19:05:32.352710 9704 ts_tablet_manager.cc:1939] T 1baaf235825346b2904b2ebc7df0c5f7 P a71340573c8c4984a53d4119e70706f4: Deleting consensus metadata
I20260620 19:05:32.354768 9152 catalog_manager.cc:4841] TS a71340573c8c4984a53d4119e70706f4 (127.1.154.129:46449): tablet 1baaf235825346b2904b2ebc7df0c5f7 (table test-table [id=4a661c65fe004fe6a8004c384f8f95c7]) successfully deleted
I20260620 19:05:32.355396 9444 tablet_service.cc:1467] Processing CreateTablet for tablet 377d9fdda7334e48bfe79c05625b48fc (DEFAULT_TABLE table=test-table [id=14e56586c1174edfb942c9ea3c438c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:32.355677 9444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377d9fdda7334e48bfe79c05625b48fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:32.355711 9304 tablet_service.cc:1467] Processing CreateTablet for tablet 377d9fdda7334e48bfe79c05625b48fc (DEFAULT_TABLE table=test-table [id=14e56586c1174edfb942c9ea3c438c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:32.355842 9678 tablet_bootstrap.cc:492] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114: Bootstrap starting.
I20260620 19:05:32.355966 9304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377d9fdda7334e48bfe79c05625b48fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:32.356802 9678 tablet_bootstrap.cc:654] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:32.358399 9679 tablet_bootstrap.cc:492] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4: Bootstrap starting.
I20260620 19:05:32.358482 9680 tablet_bootstrap.cc:492] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553: Bootstrap starting.
I20260620 19:05:32.358670 9702 ts_tablet_manager.cc:1905] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:32.359397 9680 tablet_bootstrap.cc:654] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:32.359397 9679 tablet_bootstrap.cc:654] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:32.359899 9703 ts_tablet_manager.cc:1918] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:32.360062 9703 log.cc:1196] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/1baaf235825346b2904b2ebc7df0c5f7
I20260620 19:05:32.360455 9703 ts_tablet_manager.cc:1939] T 1baaf235825346b2904b2ebc7df0c5f7 P ec0c4b27112c4d6695befd0528a61553: Deleting consensus metadata
I20260620 19:05:32.361506 9702 ts_tablet_manager.cc:1918] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:32.361632 9702 log.cc:1196] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/1baaf235825346b2904b2ebc7df0c5f7
I20260620 19:05:32.361737 9152 catalog_manager.cc:4841] TS ec0c4b27112c4d6695befd0528a61553 (127.1.154.130:40951): tablet 1baaf235825346b2904b2ebc7df0c5f7 (table test-table [id=4a661c65fe004fe6a8004c384f8f95c7]) successfully deleted
I20260620 19:05:32.361986 9702 ts_tablet_manager.cc:1939] T 1baaf235825346b2904b2ebc7df0c5f7 P 729762e4e989441cbd1c684bf6eed114: Deleting consensus metadata
I20260620 19:05:32.362972 9679 tablet_bootstrap.cc:492] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4: No bootstrap required, opened a new log
I20260620 19:05:32.363070 9679 ts_tablet_manager.cc:1397] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260620 19:05:32.363229 9151 catalog_manager.cc:4841] TS 729762e4e989441cbd1c684bf6eed114 (127.1.154.131:35867): tablet 1baaf235825346b2904b2ebc7df0c5f7 (table test-table [id=4a661c65fe004fe6a8004c384f8f95c7]) successfully deleted
I20260620 19:05:32.363903 9680 tablet_bootstrap.cc:492] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553: No bootstrap required, opened a new log
I20260620 19:05:32.363983 9680 ts_tablet_manager.cc:1397] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 19:05:32.364001 9679 raft_consensus.cc:348] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.364126 9679 raft_consensus.cc:374] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:32.364164 9679 raft_consensus.cc:729] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71340573c8c4984a53d4119e70706f4, State: Initialized, Role: FOLLOWER
I20260620 19:05:32.364300 9678 tablet_bootstrap.cc:492] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114: No bootstrap required, opened a new log
I20260620 19:05:32.364377 9678 ts_tablet_manager.cc:1397] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260620 19:05:32.364363 9679 consensus_queue.cc:260] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.364403 9680 raft_consensus.cc:348] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.364506 9680 raft_consensus.cc:374] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:32.364537 9680 raft_consensus.cc:729] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0c4b27112c4d6695befd0528a61553, State: Initialized, Role: FOLLOWER
I20260620 19:05:32.364660 9680 consensus_queue.cc:260] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.364704 9679 ts_tablet_manager.cc:1428] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 19:05:32.364899 9680 ts_tablet_manager.cc:1428] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:05:32.364874 9678 raft_consensus.cc:348] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.364987 9678 raft_consensus.cc:374] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:32.365020 9678 raft_consensus.cc:729] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729762e4e989441cbd1c684bf6eed114, State: Initialized, Role: FOLLOWER
I20260620 19:05:32.365154 9678 consensus_queue.cc:260] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.365428 9678 ts_tablet_manager.cc:1428] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260620 19:05:32.372786 9651 tablet.cc:2369] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:05:32.392002 9685 raft_consensus.cc:482] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:32.392289 9685 raft_consensus.cc:504] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.393877 9685 leader_election.cc:288] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec0c4b27112c4d6695befd0528a61553 (127.1.154.130:40951), 729762e4e989441cbd1c684bf6eed114 (127.1.154.131:35867)
I20260620 19:05:32.400354 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.394314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:32991 (local address 127.1.154.130:40951)
0620 19:05:32.394479 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.394484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.394672 (+ 188us) server_negotiation.cc:407] Connection header received
0620 19:05:32.394894 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.394898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.394950 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.394999 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:32.395587 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.396369 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.398862 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.399515 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.399658 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.399744 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.400131 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.400208 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 19:05:32.400382 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.394172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:40951 (local address 127.1.154.129:32991)
0620 19:05:32.394564 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.394580 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.394764 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.395145 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.395148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.395200 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:32.395415 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.395424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.396504 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.396507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.398669 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.398681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.399690 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.399694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.399945 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.399971 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.400097 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.400195 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":175,"spinlock_wait_cycles":37632,"thread_start_us":81,"threads_started":1}
I20260620 19:05:32.401669 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.394729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:50975 (local address 127.1.154.131:35867)
0620 19:05:32.394875 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.394880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.394896 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:05:32.394993 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.394996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.395047 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.395129 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:32.395474 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.396359 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.399848 (+ 3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.400717 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.400859 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.400908 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.401400 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.401498 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:05:32.401532 9460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377d9fdda7334e48bfe79c05625b48fc" candidate_uuid: "a71340573c8c4984a53d4119e70706f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec0c4b27112c4d6695befd0528a61553" is_pre_election: true
I20260620 19:05:32.401872 9460 raft_consensus.cc:2454] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a71340573c8c4984a53d4119e70706f4 in term 0.
I20260620 19:05:32.401698 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.394253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:35867 (local address 127.1.154.129:50975)
0620 19:05:32.394705 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.394720 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.394880 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.395150 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.395152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.395203 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:32.395364 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.395370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.396504 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.396507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.398669 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.398681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.401038 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.401042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.401184 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.401216 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.401357 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.401459 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":301,"spinlock_wait_cycles":9344,"thread_start_us":144,"threads_started":1}
I20260620 19:05:32.402509 9256 leader_election.cc:302] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a71340573c8c4984a53d4119e70706f4, ec0c4b27112c4d6695befd0528a61553; no voters:
I20260620 19:05:32.402846 9685 raft_consensus.cc:2781] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:32.402936 9685 raft_consensus.cc:482] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:32.402977 9685 raft_consensus.cc:3037] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:32.403107 9604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377d9fdda7334e48bfe79c05625b48fc" candidate_uuid: "a71340573c8c4984a53d4119e70706f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "729762e4e989441cbd1c684bf6eed114" is_pre_election: true
I20260620 19:05:32.403261 9604 raft_consensus.cc:2454] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a71340573c8c4984a53d4119e70706f4 in term 0.
I20260620 19:05:32.404130 9685 raft_consensus.cc:504] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.404543 9685 leader_election.cc:288] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [CANDIDATE]: Term 1 election: Requested vote from peers ec0c4b27112c4d6695befd0528a61553 (127.1.154.130:40951), 729762e4e989441cbd1c684bf6eed114 (127.1.154.131:35867)
I20260620 19:05:32.404878 9604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377d9fdda7334e48bfe79c05625b48fc" candidate_uuid: "a71340573c8c4984a53d4119e70706f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "729762e4e989441cbd1c684bf6eed114"
I20260620 19:05:32.404928 9460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377d9fdda7334e48bfe79c05625b48fc" candidate_uuid: "a71340573c8c4984a53d4119e70706f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec0c4b27112c4d6695befd0528a61553"
I20260620 19:05:32.405011 9604 raft_consensus.cc:3037] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:32.405043 9460 raft_consensus.cc:3037] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:32.405968 9604 raft_consensus.cc:2454] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a71340573c8c4984a53d4119e70706f4 in term 1.
I20260620 19:05:32.406247 9255 leader_election.cc:302] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 729762e4e989441cbd1c684bf6eed114, a71340573c8c4984a53d4119e70706f4; no voters:
I20260620 19:05:32.406307 9460 raft_consensus.cc:2454] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a71340573c8c4984a53d4119e70706f4 in term 1.
I20260620 19:05:32.406424 9685 raft_consensus.cc:2781] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:32.406659 9685 raft_consensus.cc:686] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [term 1 LEADER]: Becoming Leader. State: Replica: a71340573c8c4984a53d4119e70706f4, State: Running, Role: LEADER
I20260620 19:05:32.406920 9685 consensus_queue.cc:237] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.409960 9166 catalog_manager.cc:5495] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a71340573c8c4984a53d4119e70706f4 (127.1.154.129). New cstate: current_term: 1 leader_uuid: "a71340573c8c4984a53d4119e70706f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } health_report { overall_health: UNKNOWN } } }
Jun 20 19:05:32 dist-test-slave-6wrf krb5kdc[9122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982331, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.190@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
I20260620 19:05:32.449357 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.439461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:41221 (local address 127.0.0.1:35200)
0620 19:05:32.439733 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.439751 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.439860 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.440155 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.440157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.440570 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:32.440660 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.440666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.441740 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.441743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.442282 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.442289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.442875 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.442878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.442938 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.443642 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:32.443663 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:32.445314 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:32.447273 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:32.447280 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:32.447285 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:32.447522 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:32.447899 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:32.447904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:32.447906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:32.448090 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:32.448644 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:32.448653 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:32.448806 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.448933 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.449155 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":109,"threads_started":1}
I20260620 19:05:32.449501 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.439569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35200 (local address 127.1.154.190:41221)
0620 19:05:32.439707 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.439712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.439836 (+ 124us) server_negotiation.cc:407] Connection header received
0620 19:05:32.439955 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.439958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.440006 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.440108 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:32.440785 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.441635 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.442389 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.442721 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.442817 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.445457 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:32.445461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:32.445464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:32.445487 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:32.447141 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:32.447616 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:32.447619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:32.447621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:32.447706 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:32.448251 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:32.448254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:32.448256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:32.448479 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:32.448589 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.449040 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.449267 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":61}
I20260620 19:05:32.454764 9166 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:35200:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:05:32.455991 9166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:32.462813 9584 tablet_service.cc:1467] Processing CreateTablet for tablet 97bc489321ef42e283d53139710fc476 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ed2fa4ea0dd4245936f6a4130730fbd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:05:32.462814 9304 tablet_service.cc:1467] Processing CreateTablet for tablet 97bc489321ef42e283d53139710fc476 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ed2fa4ea0dd4245936f6a4130730fbd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:05:32.463222 9584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97bc489321ef42e283d53139710fc476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:32.463419 9304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97bc489321ef42e283d53139710fc476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:32.467075 9444 tablet_service.cc:1467] Processing CreateTablet for tablet 97bc489321ef42e283d53139710fc476 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ed2fa4ea0dd4245936f6a4130730fbd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:05:32.467483 9444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97bc489321ef42e283d53139710fc476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:32.467676 9678 tablet_bootstrap.cc:492] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114: Bootstrap starting.
I20260620 19:05:32.468827 9678 tablet_bootstrap.cc:654] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:32.470534 9678 tablet_bootstrap.cc:492] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114: No bootstrap required, opened a new log
I20260620 19:05:32.470628 9678 ts_tablet_manager.cc:1397] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:32.471186 9680 tablet_bootstrap.cc:492] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553: Bootstrap starting.
I20260620 19:05:32.471149 9678 raft_consensus.cc:348] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.471277 9678 raft_consensus.cc:374] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:32.471310 9678 raft_consensus.cc:729] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729762e4e989441cbd1c684bf6eed114, State: Initialized, Role: FOLLOWER
I20260620 19:05:32.471468 9678 consensus_queue.cc:260] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.471483 9679 tablet_bootstrap.cc:492] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4: Bootstrap starting.
I20260620 19:05:32.471702 9684 tablet_replica.cc:439] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } } }
I20260620 19:05:32.471863 9684 tablet_replica.cc:442] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:32.473067 9679 tablet_bootstrap.cc:654] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:32.473737 9678 ts_tablet_manager.cc:1428] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:32.473793 9680 tablet_bootstrap.cc:654] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:32.476132 9679 tablet_bootstrap.cc:492] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4: No bootstrap required, opened a new log
I20260620 19:05:32.476249 9679 ts_tablet_manager.cc:1397] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260620 19:05:32.476824 9679 raft_consensus.cc:348] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.476943 9679 raft_consensus.cc:374] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:32.476976 9679 raft_consensus.cc:729] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71340573c8c4984a53d4119e70706f4, State: Initialized, Role: FOLLOWER
I20260620 19:05:32.477082 9680 tablet_bootstrap.cc:492] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553: No bootstrap required, opened a new log
I20260620 19:05:32.477108 9679 consensus_queue.cc:260] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.477185 9680 ts_tablet_manager.cc:1397] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 19:05:32.477327 9710 tablet_replica.cc:439] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } } }
I20260620 19:05:32.477463 9710 tablet_replica.cc:442] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:32.477730 9679 ts_tablet_manager.cc:1428] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:05:32.478462 9680 raft_consensus.cc:348] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.478580 9680 raft_consensus.cc:374] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:32.478619 9680 raft_consensus.cc:729] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0c4b27112c4d6695befd0528a61553, State: Initialized, Role: FOLLOWER
I20260620 19:05:32.478760 9680 consensus_queue.cc:260] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.479010 9698 tablet_replica.cc:439] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } } }
I20260620 19:05:32.479136 9698 tablet_replica.cc:442] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:32.479364 9680 ts_tablet_manager.cc:1428] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:05:32.512751 9684 raft_consensus.cc:482] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:32.512923 9684 raft_consensus.cc:504] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.514807 9684 leader_election.cc:288] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a71340573c8c4984a53d4119e70706f4 (127.1.154.129:46449), ec0c4b27112c4d6695befd0528a61553 (127.1.154.130:40951)
I20260620 19:05:32.519644 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.515174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:46449 (local address 127.1.154.131:37659)
0620 19:05:32.515364 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.515385 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.515548 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.515963 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.515967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.516014 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:32.516158 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.516166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.517214 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.517217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.518149 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.518158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.519226 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.519230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.519326 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.519347 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.519413 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.519483 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 19:05:32.520112 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.515236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:37659 (local address 127.1.154.129:46449)
0620 19:05:32.515367 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.515374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.515658 (+ 284us) server_negotiation.cc:407] Connection header received
0620 19:05:32.515728 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.515731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.515789 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.515912 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:32.516284 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.517068 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.518258 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.518963 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.519069 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.519122 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.519947 (+ 825us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.520021 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:05:32.520534 9324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97bc489321ef42e283d53139710fc476" candidate_uuid: "729762e4e989441cbd1c684bf6eed114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71340573c8c4984a53d4119e70706f4" is_pre_election: true
I20260620 19:05:32.520694 9324 raft_consensus.cc:2454] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 729762e4e989441cbd1c684bf6eed114 in term 0.
I20260620 19:05:32.520959 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.515632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:39365 (local address 127.1.154.130:40951)
0620 19:05:32.515776 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.515783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.516052 (+ 269us) server_negotiation.cc:407] Connection header received
0620 19:05:32.516269 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.516274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.516334 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.516428 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:32.517197 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.518216 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.519624 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.520202 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.520336 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.520384 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.520710 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.520822 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 19:05:32.521062 9536 leader_election.cc:302] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 729762e4e989441cbd1c684bf6eed114, a71340573c8c4984a53d4119e70706f4; no voters:
I20260620 19:05:32.521332 9684 raft_consensus.cc:2781] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:32.521373 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.515521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:40951 (local address 127.1.154.131:39365)
0620 19:05:32.515967 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.515981 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.516104 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.516494 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.516497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.516551 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:32.517054 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.517061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.518342 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.518345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.519519 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.519527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.520447 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.520451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.520559 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.520580 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.521127 (+ 547us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.521214 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":144,"threads_started":1}
I20260620 19:05:32.521389 9684 raft_consensus.cc:482] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:32.521462 9684 raft_consensus.cc:3037] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:32.521844 9460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97bc489321ef42e283d53139710fc476" candidate_uuid: "729762e4e989441cbd1c684bf6eed114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec0c4b27112c4d6695befd0528a61553" is_pre_election: true
I20260620 19:05:32.521981 9460 raft_consensus.cc:2454] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 729762e4e989441cbd1c684bf6eed114 in term 0.
I20260620 19:05:32.522652 9684 raft_consensus.cc:504] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.523008 9684 leader_election.cc:288] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [CANDIDATE]: Term 1 election: Requested vote from peers a71340573c8c4984a53d4119e70706f4 (127.1.154.129:46449), ec0c4b27112c4d6695befd0528a61553 (127.1.154.130:40951)
I20260620 19:05:32.523356 9324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97bc489321ef42e283d53139710fc476" candidate_uuid: "729762e4e989441cbd1c684bf6eed114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71340573c8c4984a53d4119e70706f4"
I20260620 19:05:32.523376 9460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97bc489321ef42e283d53139710fc476" candidate_uuid: "729762e4e989441cbd1c684bf6eed114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec0c4b27112c4d6695befd0528a61553"
I20260620 19:05:32.523453 9324 raft_consensus.cc:3037] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:32.523506 9460 raft_consensus.cc:3037] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:32.524304 9324 raft_consensus.cc:2454] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 729762e4e989441cbd1c684bf6eed114 in term 1.
I20260620 19:05:32.524623 9536 leader_election.cc:302] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 729762e4e989441cbd1c684bf6eed114, a71340573c8c4984a53d4119e70706f4; no voters:
I20260620 19:05:32.524638 9460 raft_consensus.cc:2454] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 729762e4e989441cbd1c684bf6eed114 in term 1.
I20260620 19:05:32.524809 9684 raft_consensus.cc:2781] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:32.525064 9684 raft_consensus.cc:686] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [term 1 LEADER]: Becoming Leader. State: Replica: 729762e4e989441cbd1c684bf6eed114, State: Running, Role: LEADER
I20260620 19:05:32.525334 9684 consensus_queue.cc:237] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.526858 9722 tablet_replica.cc:439] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114: TxnStatusTablet state changed. Reason: New leader 729762e4e989441cbd1c684bf6eed114. Latest consensus state: current_term: 1 leader_uuid: "729762e4e989441cbd1c684bf6eed114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } } }
I20260620 19:05:32.526957 9722 tablet_replica.cc:442] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:05:32.527169 9724 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:05:32.527309 9724 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:05:32.527930 9166 catalog_manager.cc:5495] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 reported cstate change: term changed from 0 to 1, leader changed from <none> to 729762e4e989441cbd1c684bf6eed114 (127.1.154.131). New cstate: current_term: 1 leader_uuid: "729762e4e989441cbd1c684bf6eed114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } health_report { overall_health: HEALTHY } } }
I20260620 19:05:32.554510 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.549526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:35867 (local address 127.0.0.1:60936)
0620 19:05:32.549745 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.549759 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.549850 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.550503 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.550507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.550518 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:05:32.550657 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.550663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.551743 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.551746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.552479 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.552492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.553134 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.553138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.553266 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.553328 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:05:32.554201 (+ 873us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:05:32.554216 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.554294 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.554385 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 19:05:32.554527 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.549996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.1.154.131:35867)
0620 19:05:32.550132 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.550137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.550152 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:05:32.550225 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.550228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.550291 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.550403 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:32.550801 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.551601 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.552627 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.552985 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.553103 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.553473 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:32.554077 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:32.554188 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.554331 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.554395 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:05:32.557803 9324 raft_consensus.cc:1261] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Refusing update from remote peer 729762e4e989441cbd1c684bf6eed114: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:32.558233 9460 raft_consensus.cc:1261] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553 [term 1 FOLLOWER]: Refusing update from remote peer 729762e4e989441cbd1c684bf6eed114: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:32.558373 9722 consensus_queue.cc:1035] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:32.558954 9722 consensus_queue.cc:1035] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:32.560752 9698 tablet_replica.cc:439] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553: TxnStatusTablet state changed. Reason: New leader 729762e4e989441cbd1c684bf6eed114. Latest consensus state: current_term: 1 leader_uuid: "729762e4e989441cbd1c684bf6eed114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } } }
I20260620 19:05:32.560883 9698 tablet_replica.cc:442] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:32.561079 9710 tablet_replica.cc:439] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4: TxnStatusTablet state changed. Reason: New leader 729762e4e989441cbd1c684bf6eed114. Latest consensus state: current_term: 1 leader_uuid: "729762e4e989441cbd1c684bf6eed114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } } }
I20260620 19:05:32.561210 9710 tablet_replica.cc:442] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:32.561355 9722 tablet_replica.cc:439] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "729762e4e989441cbd1c684bf6eed114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } } }
I20260620 19:05:32.561455 9722 tablet_replica.cc:442] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:05:32.563951 9710 tablet_replica.cc:439] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "729762e4e989441cbd1c684bf6eed114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } } }
I20260620 19:05:32.564076 9710 tablet_replica.cc:442] T 97bc489321ef42e283d53139710fc476 P a71340573c8c4984a53d4119e70706f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:32.567413 9684 tablet_replica.cc:439] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "729762e4e989441cbd1c684bf6eed114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } } }
I20260620 19:05:32.567643 9684 tablet_replica.cc:442] T 97bc489321ef42e283d53139710fc476 P 729762e4e989441cbd1c684bf6eed114: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:05:32.568733 9698 tablet_replica.cc:439] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "729762e4e989441cbd1c684bf6eed114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } } }
I20260620 19:05:32.568853 9698 tablet_replica.cc:442] T 97bc489321ef42e283d53139710fc476 P ec0c4b27112c4d6695befd0528a61553: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:05:32.573204 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.566737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.1.154.129:46449)
0620 19:05:32.566864 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.566869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.566882 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:05:32.566953 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.566956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.567060 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.567174 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:32.569067 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.569835 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.571020 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.571391 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.571490 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.571718 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:32.572596 (+ 878us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:32.572802 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.572889 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.573070 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:05:32.584790 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.580445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:48119 (local address 127.1.154.131:35867)
0620 19:05:32.580568 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.580577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.581927 (+ 1350us) server_negotiation.cc:407] Connection header received
0620 19:05:32.582083 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.582086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.582132 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.582212 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:32.582446 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.582947 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.583785 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.584007 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.584108 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.584374 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:32.584446 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:32.584520 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.584604 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.584647 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:05:32.585048 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.580340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:35867 (local address 127.1.154.129:48119)
0620 19:05:32.581844 (+ 1504us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.581857 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.581978 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.582220 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.582222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.582228 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:05:32.582345 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.582356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.583045 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.583051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.583674 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.583681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.584109 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.584112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.584201 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.584248 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:05:32.584524 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:05:32.584532 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.584595 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.584664 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1357,"thread_start_us":116,"threads_started":1}
I20260620 19:05:32.593230 9460 raft_consensus.cc:1261] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553 [term 1 FOLLOWER]: Refusing update from remote peer a71340573c8c4984a53d4119e70706f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:32.593253 9604 raft_consensus.cc:1261] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Refusing update from remote peer a71340573c8c4984a53d4119e70706f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:32.593827 9710 consensus_queue.cc:1035] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:32.594084 9685 consensus_queue.cc:1035] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:32.620432 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.616469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:46449 (local address 127.1.154.131:56671)
0620 19:05:32.616643 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.616661 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.616753 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.616997 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.617000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.617006 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:05:32.617126 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.617134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.618175 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.618179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.619096 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.619106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.619675 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.619679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.619765 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.619815 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:05:32.620115 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:05:32.620120 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.620200 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.620289 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 19:05:32.620839 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.616678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:56671 (local address 127.1.154.129:46449)
0620 19:05:32.616785 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.616789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.616818 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:05:32.616861 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.616863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.616910 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.616984 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:32.617239 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.618049 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.619243 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.619518 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.619604 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.619949 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:32.620026 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:32.620083 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.620681 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.620728 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:05:32.652792 9166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:32.652992 9166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:32.655462 9166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0: Sending DeleteTablet for 3 replicas of tablet 377d9fdda7334e48bfe79c05625b48fc
I20260620 19:05:32.656143 9584 tablet_service.cc:1514] Processing DeleteTablet for tablet 377d9fdda7334e48bfe79c05625b48fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:32 UTC) from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:60926
I20260620 19:05:32.656173 9444 tablet_service.cc:1514] Processing DeleteTablet for tablet 377d9fdda7334e48bfe79c05625b48fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:32 UTC) from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:50408
I20260620 19:05:32.656330 9702 tablet_replica.cc:330] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114: stopping tablet replica
I20260620 19:05:32.656391 9703 tablet_replica.cc:330] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553: stopping tablet replica
I20260620 19:05:32.656438 9702 raft_consensus.cc:2229] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:32.656507 9703 raft_consensus.cc:2229] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:32.656562 9702 raft_consensus.cc:2258] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:32.656620 9703 raft_consensus.cc:2258] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:32.656915 9304 tablet_service.cc:1514] Processing DeleteTablet for tablet 377d9fdda7334e48bfe79c05625b48fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:32 UTC) from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:51996
I20260620 19:05:32.657092 9704 tablet_replica.cc:330] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4: stopping tablet replica
I20260620 19:05:32.657188 9704 raft_consensus.cc:2229] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:05:32.657401 9702 ts_tablet_manager.cc:1905] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:32.657488 9704 raft_consensus.cc:2258] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:32.658512 9704 ts_tablet_manager.cc:1905] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:32.660367 9702 ts_tablet_manager.cc:1918] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:05:32.660434 9702 log.cc:1196] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/377d9fdda7334e48bfe79c05625b48fc
I20260620 19:05:32.660682 9702 ts_tablet_manager.cc:1939] T 377d9fdda7334e48bfe79c05625b48fc P 729762e4e989441cbd1c684bf6eed114: Deleting consensus metadata
I20260620 19:05:32.661289 9704 ts_tablet_manager.cc:1918] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:05:32.661355 9704 log.cc:1196] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/wal/wals/377d9fdda7334e48bfe79c05625b48fc
I20260620 19:05:32.661693 9703 ts_tablet_manager.cc:1905] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:32.661887 9704 ts_tablet_manager.cc:1939] T 377d9fdda7334e48bfe79c05625b48fc P a71340573c8c4984a53d4119e70706f4: Deleting consensus metadata
I20260620 19:05:32.662688 9152 catalog_manager.cc:4841] TS a71340573c8c4984a53d4119e70706f4 (127.1.154.129:46449): tablet 377d9fdda7334e48bfe79c05625b48fc (table test-table [id=14e56586c1174edfb942c9ea3c438c7a]) successfully deleted
I20260620 19:05:32.664469 9703 ts_tablet_manager.cc:1918] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:05:32.664556 9703 log.cc:1196] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/377d9fdda7334e48bfe79c05625b48fc
I20260620 19:05:32.664804 9703 ts_tablet_manager.cc:1939] T 377d9fdda7334e48bfe79c05625b48fc P ec0c4b27112c4d6695befd0528a61553: Deleting consensus metadata
I20260620 19:05:32.665658 9152 catalog_manager.cc:4841] TS ec0c4b27112c4d6695befd0528a61553 (127.1.154.130:40951): tablet 377d9fdda7334e48bfe79c05625b48fc (table test-table [id=14e56586c1174edfb942c9ea3c438c7a]) successfully deleted
I20260620 19:05:32.666337 9151 catalog_manager.cc:4841] TS 729762e4e989441cbd1c684bf6eed114 (127.1.154.131:35867): tablet 377d9fdda7334e48bfe79c05625b48fc (table test-table [id=14e56586c1174edfb942c9ea3c438c7a]) successfully deleted
Jun 20 19:05:32 dist-test-slave-6wrf krb5kdc[9122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:05:32 dist-test-slave-6wrf krb5kdc[9122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:32 dist-test-slave-6wrf krb5kdc[9122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:05:32.699958 9216 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:32.691091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.1.154.190:41221)
0620 19:05:32.691269 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.691274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.691368 (+ 94us) server_negotiation.cc:407] Connection header received
0620 19:05:32.691568 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.691572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.691649 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.691739 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:32.692713 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.693515 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.695375 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.695726 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.695797 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.699825 (+ 4028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35210: BlockingRecv error: recv got EOF from 127.0.0.1:35210 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 19:05:32 dist-test-slave-6wrf krb5kdc[9122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM
I20260620 19:05:32.717010 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.702174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.1.154.190:41221)
0620 19:05:32.702307 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.702311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.705898 (+ 3587us) server_negotiation.cc:407] Connection header received
0620 19:05:32.706030 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.706034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.706090 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.706173 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:32.706939 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.707739 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.708846 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.709192 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.709297 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.713246 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:32.713250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:32.713253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:32.713283 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:32.715129 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:32.715664 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:32.715667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:32.715669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:32.715751 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:32.716042 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:32.716045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:32.716047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:32.716280 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:32.716410 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.716715 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.716853 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:05:32.720001 9166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:05:32.720567 9166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:32.726462 9304 tablet_service.cc:1467] Processing CreateTablet for tablet 65b53d4835ce4bc78defad0b38c78461 (DEFAULT_TABLE table=test-table [id=651a8392590b476b9bf539d6648f6739]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:32.726787 9304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b53d4835ce4bc78defad0b38c78461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:32.727329 9584 tablet_service.cc:1467] Processing CreateTablet for tablet 65b53d4835ce4bc78defad0b38c78461 (DEFAULT_TABLE table=test-table [id=651a8392590b476b9bf539d6648f6739]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:32.727591 9584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b53d4835ce4bc78defad0b38c78461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:32.729238 9679 tablet_bootstrap.cc:492] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4: Bootstrap starting.
I20260620 19:05:32.729944 9678 tablet_bootstrap.cc:492] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114: Bootstrap starting.
I20260620 19:05:32.730271 9679 tablet_bootstrap.cc:654] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:32.730894 9678 tablet_bootstrap.cc:654] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:32.732340 9444 tablet_service.cc:1467] Processing CreateTablet for tablet 65b53d4835ce4bc78defad0b38c78461 (DEFAULT_TABLE table=test-table [id=651a8392590b476b9bf539d6648f6739]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:32.732600 9444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b53d4835ce4bc78defad0b38c78461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:32.733176 9678 tablet_bootstrap.cc:492] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114: No bootstrap required, opened a new log
I20260620 19:05:32.733253 9678 ts_tablet_manager.cc:1397] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:32.733639 9678 raft_consensus.cc:348] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.733785 9678 raft_consensus.cc:374] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:32.733815 9678 raft_consensus.cc:729] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729762e4e989441cbd1c684bf6eed114, State: Initialized, Role: FOLLOWER
I20260620 19:05:32.733924 9678 consensus_queue.cc:260] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.734150 9678 ts_tablet_manager.cc:1428] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:05:32.734942 9680 tablet_bootstrap.cc:492] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553: Bootstrap starting.
I20260620 19:05:32.735694 9680 tablet_bootstrap.cc:654] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:32.736795 9679 tablet_bootstrap.cc:492] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4: No bootstrap required, opened a new log
I20260620 19:05:32.736886 9679 ts_tablet_manager.cc:1397] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260620 19:05:32.737315 9679 raft_consensus.cc:348] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.737417 9679 raft_consensus.cc:374] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:32.737447 9679 raft_consensus.cc:729] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71340573c8c4984a53d4119e70706f4, State: Initialized, Role: FOLLOWER
I20260620 19:05:32.737474 9680 tablet_bootstrap.cc:492] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553: No bootstrap required, opened a new log
I20260620 19:05:32.737532 9680 ts_tablet_manager.cc:1397] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 19:05:32.737558 9679 consensus_queue.cc:260] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.737848 9679 ts_tablet_manager.cc:1428] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:05:32.737987 9680 raft_consensus.cc:348] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.738104 9680 raft_consensus.cc:374] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:32.738144 9680 raft_consensus.cc:729] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0c4b27112c4d6695befd0528a61553, State: Initialized, Role: FOLLOWER
I20260620 19:05:32.738262 9680 consensus_queue.cc:260] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.738559 9680 ts_tablet_manager.cc:1428] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:05:32.795017 9698 raft_consensus.cc:482] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:32.795203 9698 raft_consensus.cc:504] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.795672 9698 leader_election.cc:288] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a71340573c8c4984a53d4119e70706f4 (127.1.154.129:46449), 729762e4e989441cbd1c684bf6eed114 (127.1.154.131:35867)
I20260620 19:05:32.796092 9324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b53d4835ce4bc78defad0b38c78461" candidate_uuid: "ec0c4b27112c4d6695befd0528a61553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71340573c8c4984a53d4119e70706f4" is_pre_election: true
I20260620 19:05:32.796137 9604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b53d4835ce4bc78defad0b38c78461" candidate_uuid: "ec0c4b27112c4d6695befd0528a61553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "729762e4e989441cbd1c684bf6eed114" is_pre_election: true
I20260620 19:05:32.796241 9324 raft_consensus.cc:2454] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec0c4b27112c4d6695befd0528a61553 in term 0.
I20260620 19:05:32.796258 9604 raft_consensus.cc:2454] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec0c4b27112c4d6695befd0528a61553 in term 0.
I20260620 19:05:32.796516 9396 leader_election.cc:302] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a71340573c8c4984a53d4119e70706f4, ec0c4b27112c4d6695befd0528a61553; no voters:
I20260620 19:05:32.796666 9698 raft_consensus.cc:2781] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:32.796728 9698 raft_consensus.cc:482] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:32.796765 9698 raft_consensus.cc:3037] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:32.797858 9698 raft_consensus.cc:504] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.798210 9698 leader_election.cc:288] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [CANDIDATE]: Term 1 election: Requested vote from peers a71340573c8c4984a53d4119e70706f4 (127.1.154.129:46449), 729762e4e989441cbd1c684bf6eed114 (127.1.154.131:35867)
I20260620 19:05:32.798503 9324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b53d4835ce4bc78defad0b38c78461" candidate_uuid: "ec0c4b27112c4d6695befd0528a61553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71340573c8c4984a53d4119e70706f4"
I20260620 19:05:32.798633 9324 raft_consensus.cc:3037] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:32.798635 9604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b53d4835ce4bc78defad0b38c78461" candidate_uuid: "ec0c4b27112c4d6695befd0528a61553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "729762e4e989441cbd1c684bf6eed114"
I20260620 19:05:32.798785 9604 raft_consensus.cc:3037] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:32.799701 9324 raft_consensus.cc:2454] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec0c4b27112c4d6695befd0528a61553 in term 1.
I20260620 19:05:32.800019 9396 leader_election.cc:302] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a71340573c8c4984a53d4119e70706f4, ec0c4b27112c4d6695befd0528a61553; no voters:
I20260620 19:05:32.800037 9604 raft_consensus.cc:2454] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec0c4b27112c4d6695befd0528a61553 in term 1.
I20260620 19:05:32.800192 9698 raft_consensus.cc:2781] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:32.800285 9698 raft_consensus.cc:686] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [term 1 LEADER]: Becoming Leader. State: Replica: ec0c4b27112c4d6695befd0528a61553, State: Running, Role: LEADER
I20260620 19:05:32.800416 9698 consensus_queue.cc:237] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.802141 9167 catalog_manager.cc:5495] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec0c4b27112c4d6695befd0528a61553 (127.1.154.130). New cstate: current_term: 1 leader_uuid: "ec0c4b27112c4d6695befd0528a61553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } health_report { overall_health: UNKNOWN } } }
I20260620 19:05:32.826489 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.822063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.1.154.130:40951)
0620 19:05:32.822190 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.822194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.822402 (+ 208us) server_negotiation.cc:407] Connection header received
0620 19:05:32.822526 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.822530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.822601 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.822693 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:32.823108 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.823733 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.825225 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.825439 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.825522 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.825950 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:32.826068 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:32.826137 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.826269 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.826326 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:05:32.828619 9324 raft_consensus.cc:1261] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Refusing update from remote peer ec0c4b27112c4d6695befd0528a61553: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:32.828629 9604 raft_consensus.cc:1261] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Refusing update from remote peer ec0c4b27112c4d6695befd0528a61553: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:32.829006 9686 consensus_queue.cc:1035] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:32.829312 9686 consensus_queue.cc:1035] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:32.839733 9165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:32.839884 9165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:32.842008 9165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0: Sending DeleteTablet for 3 replicas of tablet 65b53d4835ce4bc78defad0b38c78461
I20260620 19:05:32.842590 9444 tablet_service.cc:1514] Processing DeleteTablet for tablet 65b53d4835ce4bc78defad0b38c78461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:32 UTC) from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:50408
I20260620 19:05:32.842803 9584 tablet_service.cc:1514] Processing DeleteTablet for tablet 65b53d4835ce4bc78defad0b38c78461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:32 UTC) from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:60926
I20260620 19:05:32.842993 9703 tablet_replica.cc:330] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553: stopping tablet replica
I20260620 19:05:32.843132 9703 raft_consensus.cc:2229] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:05:32.843336 9703 raft_consensus.cc:2258] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:32.843321 9165 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:05:32.843523 9702 tablet_replica.cc:330] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114: stopping tablet replica
I20260620 19:05:32.843650 9702 raft_consensus.cc:2229] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 19:05:32.843735 9165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:32.843766 9702 raft_consensus.cc:2258] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:32.843760 9304 tablet_service.cc:1514] Processing DeleteTablet for tablet 65b53d4835ce4bc78defad0b38c78461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:32 UTC) from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:51996
I20260620 19:05:32.843943 9704 tablet_replica.cc:330] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4: stopping tablet replica
I20260620 19:05:32.844044 9704 raft_consensus.cc:2229] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:32.844182 9704 raft_consensus.cc:2258] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:32.844425 9703 ts_tablet_manager.cc:1905] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:32.844794 9704 ts_tablet_manager.cc:1905] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:32.844956 9702 ts_tablet_manager.cc:1905] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:32.847043 9704 ts_tablet_manager.cc:1918] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:32.847043 9703 ts_tablet_manager.cc:1918] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:32.847108 9703 log.cc:1196] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/65b53d4835ce4bc78defad0b38c78461
I20260620 19:05:32.847121 9704 log.cc:1196] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-0/wal/wals/65b53d4835ce4bc78defad0b38c78461
I20260620 19:05:32.847364 9702 ts_tablet_manager.cc:1918] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:32.847376 9704 ts_tablet_manager.cc:1939] T 65b53d4835ce4bc78defad0b38c78461 P a71340573c8c4984a53d4119e70706f4: Deleting consensus metadata
I20260620 19:05:32.847398 9703 ts_tablet_manager.cc:1939] T 65b53d4835ce4bc78defad0b38c78461 P ec0c4b27112c4d6695befd0528a61553: Deleting consensus metadata
I20260620 19:05:32.847419 9702 log.cc:1196] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/65b53d4835ce4bc78defad0b38c78461
I20260620 19:05:32.847658 9702 ts_tablet_manager.cc:1939] T 65b53d4835ce4bc78defad0b38c78461 P 729762e4e989441cbd1c684bf6eed114: Deleting consensus metadata
I20260620 19:05:32.848289 9151 catalog_manager.cc:4841] TS 729762e4e989441cbd1c684bf6eed114 (127.1.154.131:35867): tablet 65b53d4835ce4bc78defad0b38c78461 (table test-table [id=651a8392590b476b9bf539d6648f6739]) successfully deleted
I20260620 19:05:32.848397 9152 catalog_manager.cc:4841] TS ec0c4b27112c4d6695befd0528a61553 (127.1.154.130:40951): tablet 65b53d4835ce4bc78defad0b38c78461 (table test-table [id=651a8392590b476b9bf539d6648f6739]) successfully deleted
I20260620 19:05:32.848536 9152 catalog_manager.cc:4841] TS a71340573c8c4984a53d4119e70706f4 (127.1.154.129:46449): tablet 65b53d4835ce4bc78defad0b38c78461 (table test-table [id=651a8392590b476b9bf539d6648f6739]) successfully deleted
I20260620 19:05:32.850204 9304 tablet_service.cc:1467] Processing CreateTablet for tablet 195f927005da4048ad590596c9f55091 (DEFAULT_TABLE table=test-table [id=8a0a1105f341442d86720b6262fcd4f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:32.850292 9444 tablet_service.cc:1467] Processing CreateTablet for tablet 195f927005da4048ad590596c9f55091 (DEFAULT_TABLE table=test-table [id=8a0a1105f341442d86720b6262fcd4f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:32.850389 9584 tablet_service.cc:1467] Processing CreateTablet for tablet 195f927005da4048ad590596c9f55091 (DEFAULT_TABLE table=test-table [id=8a0a1105f341442d86720b6262fcd4f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:32.850469 9304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 195f927005da4048ad590596c9f55091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:32.850736 9444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 195f927005da4048ad590596c9f55091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:32.850849 9584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 195f927005da4048ad590596c9f55091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:32.852347 9679 tablet_bootstrap.cc:492] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4: Bootstrap starting.
I20260620 19:05:32.853055 9679 tablet_bootstrap.cc:654] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:32.853957 9678 tablet_bootstrap.cc:492] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114: Bootstrap starting.
I20260620 19:05:32.853962 9680 tablet_bootstrap.cc:492] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553: Bootstrap starting.
I20260620 19:05:32.854123 9679 tablet_bootstrap.cc:492] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4: No bootstrap required, opened a new log
I20260620 19:05:32.854189 9679 ts_tablet_manager.cc:1397] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:05:32.854611 9679 raft_consensus.cc:348] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.854722 9678 tablet_bootstrap.cc:654] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:32.854745 9679 raft_consensus.cc:374] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:32.854902 9680 tablet_bootstrap.cc:654] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:32.854920 9679 raft_consensus.cc:729] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71340573c8c4984a53d4119e70706f4, State: Initialized, Role: FOLLOWER
I20260620 19:05:32.855330 9679 consensus_queue.cc:260] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.855872 9679 ts_tablet_manager.cc:1428] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:05:32.856060 9680 tablet_bootstrap.cc:492] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553: No bootstrap required, opened a new log
I20260620 19:05:32.856176 9680 ts_tablet_manager.cc:1397] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:05:32.856496 9678 tablet_bootstrap.cc:492] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114: No bootstrap required, opened a new log
I20260620 19:05:32.856539 9680 raft_consensus.cc:348] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.856683 9680 raft_consensus.cc:374] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:32.856705 9678 ts_tablet_manager.cc:1397] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 19:05:32.856719 9680 raft_consensus.cc:729] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0c4b27112c4d6695befd0528a61553, State: Initialized, Role: FOLLOWER
I20260620 19:05:32.856856 9680 consensus_queue.cc:260] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.857115 9680 ts_tablet_manager.cc:1428] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:05:32.857295 9678 raft_consensus.cc:348] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.857422 9678 raft_consensus.cc:374] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:32.857455 9678 raft_consensus.cc:729] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729762e4e989441cbd1c684bf6eed114, State: Initialized, Role: FOLLOWER
I20260620 19:05:32.857733 9678 consensus_queue.cc:260] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.858544 9678 ts_tablet_manager.cc:1428] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260620 19:05:32.860971 9371 tablet.cc:2369] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:05:32.865417 9511 tablet.cc:2369] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:05:32.942034 9686 raft_consensus.cc:482] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:32.942214 9686 raft_consensus.cc:504] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.942835 9686 leader_election.cc:288] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a71340573c8c4984a53d4119e70706f4 (127.1.154.129:46449), 729762e4e989441cbd1c684bf6eed114 (127.1.154.131:35867)
I20260620 19:05:32.943367 9604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195f927005da4048ad590596c9f55091" candidate_uuid: "ec0c4b27112c4d6695befd0528a61553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "729762e4e989441cbd1c684bf6eed114" is_pre_election: true
I20260620 19:05:32.943368 9324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195f927005da4048ad590596c9f55091" candidate_uuid: "ec0c4b27112c4d6695befd0528a61553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71340573c8c4984a53d4119e70706f4" is_pre_election: true
I20260620 19:05:32.943502 9604 raft_consensus.cc:2454] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec0c4b27112c4d6695befd0528a61553 in term 0.
I20260620 19:05:32.943535 9324 raft_consensus.cc:2454] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec0c4b27112c4d6695befd0528a61553 in term 0.
I20260620 19:05:32.943876 9396 leader_election.cc:302] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a71340573c8c4984a53d4119e70706f4, ec0c4b27112c4d6695befd0528a61553; no voters:
I20260620 19:05:32.944061 9686 raft_consensus.cc:2781] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:32.944123 9686 raft_consensus.cc:482] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:32.944195 9686 raft_consensus.cc:3037] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:32.945300 9686 raft_consensus.cc:504] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.945724 9686 leader_election.cc:288] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [CANDIDATE]: Term 1 election: Requested vote from peers a71340573c8c4984a53d4119e70706f4 (127.1.154.129:46449), 729762e4e989441cbd1c684bf6eed114 (127.1.154.131:35867)
I20260620 19:05:32.946054 9324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195f927005da4048ad590596c9f55091" candidate_uuid: "ec0c4b27112c4d6695befd0528a61553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71340573c8c4984a53d4119e70706f4"
I20260620 19:05:32.946197 9324 raft_consensus.cc:3037] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:32.946447 9604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195f927005da4048ad590596c9f55091" candidate_uuid: "ec0c4b27112c4d6695befd0528a61553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "729762e4e989441cbd1c684bf6eed114"
I20260620 19:05:32.946660 9604 raft_consensus.cc:3037] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:32.947202 9324 raft_consensus.cc:2454] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec0c4b27112c4d6695befd0528a61553 in term 1.
I20260620 19:05:32.947479 9396 leader_election.cc:302] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a71340573c8c4984a53d4119e70706f4, ec0c4b27112c4d6695befd0528a61553; no voters:
I20260620 19:05:32.947656 9686 raft_consensus.cc:2781] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:32.947758 9686 raft_consensus.cc:686] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [term 1 LEADER]: Becoming Leader. State: Replica: ec0c4b27112c4d6695befd0528a61553, State: Running, Role: LEADER
I20260620 19:05:32.947887 9686 consensus_queue.cc:237] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } }
I20260620 19:05:32.948051 9604 raft_consensus.cc:2454] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec0c4b27112c4d6695befd0528a61553 in term 1.
I20260620 19:05:32.949787 9165 catalog_manager.cc:5495] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec0c4b27112c4d6695befd0528a61553 (127.1.154.130). New cstate: current_term: 1 leader_uuid: "ec0c4b27112c4d6695befd0528a61553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec0c4b27112c4d6695befd0528a61553" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 40951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 } health_report { overall_health: UNKNOWN } } }
I20260620 19:05:32.998530 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.994978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:35867 (local address 127.1.154.130:58163)
0620 19:05:32.995386 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:32.995409 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:32.995544 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:32.995791 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:32.995794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:32.995802 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:05:32.995916 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.995921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.996648 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.996653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.997295 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:32.997303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.997808 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.997811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:32.997894 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.997934 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:05:32.998235 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:05:32.998243 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:05:32.998308 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:05:32.998400 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":112,"threads_started":1}
I20260620 19:05:32.998528 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:32.995153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:58163 (local address 127.1.154.131:35867)
0620 19:05:32.995357 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:32.995362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:32.995484 (+ 122us) server_negotiation.cc:407] Connection header received
0620 19:05:32.995644 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:32.995647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:32.995698 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:32.995780 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:32.996015 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.996544 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.997413 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:32.997693 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:32.997797 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:32.998066 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:32.998151 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:32.998227 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:32.998318 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 19:05:32.998376 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 19:05:33.006155 9324 raft_consensus.cc:1261] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Refusing update from remote peer ec0c4b27112c4d6695befd0528a61553: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:33.006155 9604 raft_consensus.cc:1261] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Refusing update from remote peer ec0c4b27112c4d6695befd0528a61553: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:33.006606 9686 consensus_queue.cc:1035] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a71340573c8c4984a53d4119e70706f4" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 46449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:33.006795 9696 consensus_queue.cc:1035] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729762e4e989441cbd1c684bf6eed114" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:33.028821 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:33.024694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:59591 (local address 127.1.154.130:40951)
0620 19:05:33.024835 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:33.024841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:33.024858 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:05:33.024932 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:33.024936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:33.024997 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:33.025087 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:33.025287 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:33.026194 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:33.027676 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:33.027965 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:33.028096 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:33.028361 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:33.028469 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:33.028537 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:33.028622 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 19:05:33.028676 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:05:33.028822 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:33.024619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:40951 (local address 127.1.154.131:59591)
0620 19:05:33.024761 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:33.024779 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:33.024855 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:33.025092 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:33.025094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:33.025100 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:05:33.025185 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:33.025189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:33.026390 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:33.026396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:33.027525 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:33.027534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:33.028096 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:33.028099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:33.028190 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:33.028228 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:05:33.028540 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:05:33.028548 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:05:33.028614 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:05:33.028697 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 19:05:33.051790 9165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:33.051957 9165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:33.055156 9165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7a9f01fb7c9447597e059f0b26c14f0: Sending DeleteTablet for 3 replicas of tablet 195f927005da4048ad590596c9f55091
I20260620 19:05:33.055974 9584 tablet_service.cc:1514] Processing DeleteTablet for tablet 195f927005da4048ad590596c9f55091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:33 UTC) from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:60926
I20260620 19:05:33.056059 9304 tablet_service.cc:1514] Processing DeleteTablet for tablet 195f927005da4048ad590596c9f55091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:33 UTC) from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:51996
I20260620 19:05:33.056080 9444 tablet_service.cc:1514] Processing DeleteTablet for tablet 195f927005da4048ad590596c9f55091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:33 UTC) from {username='oryx', principal='oryx/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:50408
I20260620 19:05:33.056162 9702 tablet_replica.cc:330] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114: stopping tablet replica
I20260620 19:05:33.056280 9704 tablet_replica.cc:330] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4: stopping tablet replica
I20260620 19:05:33.056337 9702 raft_consensus.cc:2229] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:33.056398 9704 raft_consensus.cc:2229] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:33.056468 9702 raft_consensus.cc:2258] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:33.056502 9704 raft_consensus.cc:2258] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:33.056758 9703 tablet_replica.cc:330] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553: stopping tablet replica
I20260620 19:05:33.056895 9703 raft_consensus.cc:2229] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:05:33.057106 9703 raft_consensus.cc:2258] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:33.057394 9704 ts_tablet_manager.cc:1905] T 195f927005da4048ad590596c9f55091 P a71340573c8c4984a53d4119e70706f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:33.057449 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 9239
I20260620 19:05:33.059077 9702 ts_tablet_manager.cc:1905] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:33.059504 9703 ts_tablet_manager.cc:1905] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:33.061867 9703 ts_tablet_manager.cc:1918] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:05:33.061968 9703 log.cc:1196] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/195f927005da4048ad590596c9f55091
I20260620 19:05:33.062230 9703 ts_tablet_manager.cc:1939] T 195f927005da4048ad590596c9f55091 P ec0c4b27112c4d6695befd0528a61553: Deleting consensus metadata
I20260620 19:05:33.062309 9702 ts_tablet_manager.cc:1918] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:05:33.062407 9702 log.cc:1196] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/195f927005da4048ad590596c9f55091
I20260620 19:05:33.062665 9702 ts_tablet_manager.cc:1939] T 195f927005da4048ad590596c9f55091 P 729762e4e989441cbd1c684bf6eed114: Deleting consensus metadata
I20260620 19:05:33.063110 9152 catalog_manager.cc:4841] TS ec0c4b27112c4d6695befd0528a61553 (127.1.154.130:40951): tablet 195f927005da4048ad590596c9f55091 (table test-table [id=8a0a1105f341442d86720b6262fcd4f3]) successfully deleted
I20260620 19:05:33.063448 9151 catalog_manager.cc:4841] TS 729762e4e989441cbd1c684bf6eed114 (127.1.154.131:35867): tablet 195f927005da4048ad590596c9f55091 (table test-table [id=8a0a1105f341442d86720b6262fcd4f3]) successfully deleted
I20260620 19:05:33.067008 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 9379
W20260620 19:05:33.067027 9152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:46449 (error 108)
I20260620 19:05:33.069356 9768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:33.068510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:46449 (local address 127.0.0.1:52010)
0620 19:05:33.069048 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:33.069110 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:46449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":60,"threads_started":1}
W20260620 19:05:33.069525 9152 catalog_manager.cc:4584] TS a71340573c8c4984a53d4119e70706f4 (127.1.154.129:46449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 195f927005da4048ad590596c9f55091: Network error: Client connection negotiation failed: client connection to 127.1.154.129:46449: connect: Connection refused (error 111)
I20260620 19:05:33.074820 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 9519
I20260620 19:05:33.084123 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 9135
2026-06-20T19:05:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3209 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:05:33 dist-test-slave-6wrf krb5kdc[9771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:05:33 dist-test-slave-6wrf krb5kdc[9771](info): set up 2 sockets
Jun 20 19:05:33 dist-test-slave-6wrf krb5kdc[9771](info): commencing operation
krb5kdc: starting...
W20260620 19:05:34.182777 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:05:34 dist-test-slave-6wrf krb5kdc[9771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:05:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:05:34.321768 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:33697
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:46625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:33697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:34.442353 9784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:34.442728 9784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:34.442801 9784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:34.446990 9784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:34.447070 9784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:34.447106 9784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:34.447160 9784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:34.447196 9784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:34.452805 9784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:46625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:33697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:33697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:34.454703 9784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:34.456190 9784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:34.465533 9790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:34.465543 9789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:34.465672 9792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:34.467551 9784 server_base.cc:1034] running on GCE node
I20260620 19:05:34.468470 9784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:34.470017 9784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:34.471310 9784 hybrid_clock.cc:648] HybridClock initialized: now 1781982334471219 us; error 140 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T19:05:34Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1392 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:05:34 dist-test-slave-6wrf krb5kdc[9797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:05:34 dist-test-slave-6wrf krb5kdc[9797](info): set up 2 sockets
Jun 20 19:05:34 dist-test-slave-6wrf krb5kdc[9797](info): commencing operation
krb5kdc: starting...
W20260620 19:05:35.562695 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:05:35 dist-test-slave-6wrf krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:05:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:05:35.701349 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:39569
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:39569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:35.828290 9810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:35.828575 9810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:35.828616 9810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:35.832758 9810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:35.832835 9810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:35.832854 9810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:35.832872 9810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:35.832887 9810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:35.840229 9810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:39569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:39569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:35.841862 9810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:35.843206 9810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:35.854765 9815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:35.854766 9818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:35.854846 9816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:35.855369 9810 server_base.cc:1034] running on GCE node
I20260620 19:05:35.856434 9810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:35.858498 9810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:35.859750 9810 hybrid_clock.cc:648] HybridClock initialized: now 1781982335859733 us; error 36 us; skew 500 ppm
Jun 20 19:05:35 dist-test-slave-6wrf krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:35.866806 9810 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:05:35.867885 9810 webserver.cc:460] Webserver started at http://127.1.154.190:40175/ using document root <none> and password file <none>
I20260620 19:05:35.868479 9810 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:35.868569 9810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:35.868805 9810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:35.871573 9810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "376d3c57ac594da78966f8fce8ae8d17"
format_stamp: "Formatted at 2026-06-20 19:05:35 on dist-test-slave-6wrf"
server_key: "46b8d2b868d784fa3d5a62f02157ce25"
server_key_iv: "633278282d5f74d282babbaabe7a2a60"
server_key_version: "encryptionkey@0"
I20260620 19:05:35.872422 9810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "376d3c57ac594da78966f8fce8ae8d17"
format_stamp: "Formatted at 2026-06-20 19:05:35 on dist-test-slave-6wrf"
server_key: "46b8d2b868d784fa3d5a62f02157ce25"
server_key_iv: "633278282d5f74d282babbaabe7a2a60"
server_key_version: "encryptionkey@0"
I20260620 19:05:35.877732 9810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 19:05:35.881371 9825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:35.882921 9810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:35.883121 9810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "376d3c57ac594da78966f8fce8ae8d17"
format_stamp: "Formatted at 2026-06-20 19:05:35 on dist-test-slave-6wrf"
server_key: "46b8d2b868d784fa3d5a62f02157ce25"
server_key_iv: "633278282d5f74d282babbaabe7a2a60"
server_key_version: "encryptionkey@0"
I20260620 19:05:35.883270 9810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:35.899344 9810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:35.903992 9810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:35.904242 9810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:35.914259 9810 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:39569
I20260620 19:05:35.914264 9877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:39569 every 8 connection(s)
I20260620 19:05:35.915582 9810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:35.919224 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 9810
I20260620 19:05:35.919292 9878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:35.919404 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:35.919747 1642 external_mini_cluster.cc:1428] Setting key 6c92f89242fdaed0177048da0b7de40f
I20260620 19:05:35.926465 9878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17: Bootstrap starting.
I20260620 19:05:35.929462 9878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17: Neither blocks nor log segments found. Creating new log.
Jun 20 19:05:35 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:35.930359 9878 log.cc:824] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:35.932981 9878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17: No bootstrap required, opened a new log
I20260620 19:05:35.937115 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:35.921271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.1.154.190:39569)
0620 19:05:35.921742 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:35.921774 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:35.921836 (+ 62us) server_negotiation.cc:407] Connection header received
0620 19:05:35.922678 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:35.922696 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:35.923025 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:35.923341 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:35.924374 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:35.925964 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:35.927216 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:35.927769 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:35.927906 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:35.930908 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:35.930913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:35.930930 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:35.930958 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:35.933081 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:35.934016 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:35.934022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:35.934028 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:35.934175 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:35.934567 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:35.934570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:35.934572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:35.935136 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:35.935382 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:35.935778 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 19:05:35.936105 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":153,"threads_started":1}
I20260620 19:05:35.937733 9878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376d3c57ac594da78966f8fce8ae8d17" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39569 } }
I20260620 19:05:35.938005 9878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:35.938059 9878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376d3c57ac594da78966f8fce8ae8d17, State: Initialized, Role: FOLLOWER
I20260620 19:05:35.939477 9878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376d3c57ac594da78966f8fce8ae8d17" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39569 } }
I20260620 19:05:35.939613 9878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:35.939654 9878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:35.939725 9878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:35.940999 9878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376d3c57ac594da78966f8fce8ae8d17" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39569 } }
I20260620 19:05:35.941555 9878 leader_election.cc:302] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 376d3c57ac594da78966f8fce8ae8d17; no voters:
I20260620 19:05:35.942037 9878 leader_election.cc:288] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:35.942183 9883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:35.942500 9883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 1 LEADER]: Becoming Leader. State: Replica: 376d3c57ac594da78966f8fce8ae8d17, State: Running, Role: LEADER
I20260620 19:05:35.942857 9883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376d3c57ac594da78966f8fce8ae8d17" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39569 } }
I20260620 19:05:35.943814 9878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:35.945086 9885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 376d3c57ac594da78966f8fce8ae8d17. Latest consensus state: current_term: 1 leader_uuid: "376d3c57ac594da78966f8fce8ae8d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376d3c57ac594da78966f8fce8ae8d17" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39569 } } }
I20260620 19:05:35.945253 9885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:35.945524 9884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "376d3c57ac594da78966f8fce8ae8d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376d3c57ac594da78966f8fce8ae8d17" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39569 } } }
I20260620 19:05:35.945645 9884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:35.945983 9892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:35.949133 9892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:35.956246 9892 catalog_manager.cc:1347] Generated new cluster ID: 00ec15c3b21146e28a8c1deb79047ca5
I20260620 19:05:35.956338 9892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:35.968348 9892 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:35.969374 9892 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:35.975548 9892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17: Generated new TSK 0
I20260620 19:05:35.976311 9892 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:05:36.009392 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39569
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:36.130587 9904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:36.130993 9904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:36.131047 9904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:36.137462 9904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:36.137624 9904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:36.137778 9904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:05:36.145809 9904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:36.147012 9904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:36.148062 9904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:36.158419 9909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:36.158461 9910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:36.158519 9912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:36.158571 9904 server_base.cc:1034] running on GCE node
I20260620 19:05:36.159078 9904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:36.159741 9904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:36.160940 9904 hybrid_clock.cc:648] HybridClock initialized: now 1781982336160845 us; error 110 us; skew 500 ppm
Jun 20 19:05:36 dist-test-slave-6wrf krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:36.163915 9904 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:05:36.164883 9904 webserver.cc:460] Webserver started at http://127.1.154.129:45037/ using document root <none> and password file <none>
I20260620 19:05:36.165443 9904 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:36.165539 9904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:36.165779 9904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:36.167838 9904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "af7da3feb0bc446db1404085c166e8d1"
format_stamp: "Formatted at 2026-06-20 19:05:36 on dist-test-slave-6wrf"
server_key: "536b8018af687065e95494d327a37bae"
server_key_iv: "c1b2f14edd2b0db3c3682de30f96e4ba"
server_key_version: "encryptionkey@0"
I20260620 19:05:36.168403 9904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "af7da3feb0bc446db1404085c166e8d1"
format_stamp: "Formatted at 2026-06-20 19:05:36 on dist-test-slave-6wrf"
server_key: "536b8018af687065e95494d327a37bae"
server_key_iv: "c1b2f14edd2b0db3c3682de30f96e4ba"
server_key_version: "encryptionkey@0"
I20260620 19:05:36.172225 9904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:05:36.175519 9919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:36.177546 9904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:05:36.177733 9904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "af7da3feb0bc446db1404085c166e8d1"
format_stamp: "Formatted at 2026-06-20 19:05:36 on dist-test-slave-6wrf"
server_key: "536b8018af687065e95494d327a37bae"
server_key_iv: "c1b2f14edd2b0db3c3682de30f96e4ba"
server_key_version: "encryptionkey@0"
I20260620 19:05:36.177832 9904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:36.208127 9904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:36.211596 9904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:36.211752 9904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:36.212273 9904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:36.213044 9904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:36.213094 9904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:36.213135 9904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:36.213150 9904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:36.222241 9904 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:40957
I20260620 19:05:36.222266 10032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:40957 every 8 connection(s)
I20260620 19:05:36.223114 9904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:05:36.227432 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 9904
I20260620 19:05:36.227598 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:05:36.227891 1642 external_mini_cluster.cc:1428] Setting key 7941aa3285425a4fc37ebef90d895184
Jun 20 19:05:36 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
I20260620 19:05:36.239040 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:36.224848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:41791 (local address 127.1.154.190:39569)
0620 19:05:36.225038 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:36.225042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:36.225837 (+ 795us) server_negotiation.cc:407] Connection header received
0620 19:05:36.227120 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:36.227123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:36.227178 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:36.227279 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:36.228601 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:36.229162 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:36.231323 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:36.231567 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:36.231678 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:36.234764 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:36.234769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:36.234772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:36.234806 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:36.236704 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:36.237362 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:36.237365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:36.237368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:36.237450 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:36.237826 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:36.237829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:36.237831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:36.238021 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:36.238136 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:36.238710 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 19:05:36.238851 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:05:36.239830 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:36.225194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:41791)
0620 19:05:36.225697 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:36.225738 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:36.226910 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:36.227439 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:36.227445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:36.227860 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:36.228446 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:36.228466 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:36.229718 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:36.229725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:36.231198 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:36.231207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:36.231711 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:36.231715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:36.231890 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:36.232494 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:36.232517 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:36.234489 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:36.236920 (+ 2431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:36.236927 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:36.236944 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:36.237242 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:36.237544 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:36.237547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:36.237549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:36.237726 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:36.238132 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:36.238138 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:36.238463 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 19:05:36.238654 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 19:05:36.238992 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
I20260620 19:05:36.241349 10033 heartbeater.cc:347] Connected to a master server at 127.1.154.190:39569
I20260620 19:05:36.241671 10033 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:36.242208 10033 heartbeater.cc:511] Master 127.1.154.190:39569 requested a full tablet report, sending...
I20260620 19:05:36.244189 9842 ts_manager.cc:194] Registered new tserver with Master: af7da3feb0bc446db1404085c166e8d1 (127.1.154.129:40957)
I20260620 19:05:36.245384 9842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:41791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:05:36.261973 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39569
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:36.394680 10038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:36.394977 10038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:36.395054 10038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:36.399073 10038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:36.399163 10038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:36.399250 10038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:36.404544 10038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:36.405646 10038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:36.406668 10038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:36.416837 10046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:36.416843 10043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:36.416970 10038 server_base.cc:1034] running on GCE node
W20260620 19:05:36.417260 10044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:36.418216 10038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:36.418938 10038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:36.420171 10038 hybrid_clock.cc:648] HybridClock initialized: now 1781982336420137 us; error 44 us; skew 500 ppm
Jun 20 19:05:36 dist-test-slave-6wrf krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:36.423496 10038 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:05:36.424386 10038 webserver.cc:460] Webserver started at http://127.1.154.130:44585/ using document root <none> and password file <none>
I20260620 19:05:36.424955 10038 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:36.425042 10038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:36.425254 10038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:36.427312 10038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "a4504fdf4a56441e8081a65813719a0f"
format_stamp: "Formatted at 2026-06-20 19:05:36 on dist-test-slave-6wrf"
server_key: "aa0d009caa1f5d972328c699cb2ee052"
server_key_iv: "a32310d509d7ae44de1518990789100f"
server_key_version: "encryptionkey@0"
I20260620 19:05:36.427862 10038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "a4504fdf4a56441e8081a65813719a0f"
format_stamp: "Formatted at 2026-06-20 19:05:36 on dist-test-slave-6wrf"
server_key: "aa0d009caa1f5d972328c699cb2ee052"
server_key_iv: "a32310d509d7ae44de1518990789100f"
server_key_version: "encryptionkey@0"
I20260620 19:05:36.431593 10038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:05:36.434130 10053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:36.435940 10038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:36.436177 10038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "a4504fdf4a56441e8081a65813719a0f"
format_stamp: "Formatted at 2026-06-20 19:05:36 on dist-test-slave-6wrf"
server_key: "aa0d009caa1f5d972328c699cb2ee052"
server_key_iv: "a32310d509d7ae44de1518990789100f"
server_key_version: "encryptionkey@0"
I20260620 19:05:36.436491 10038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:36.453439 10038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:36.457021 10038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:36.457232 10038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:36.457940 10038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:36.458812 10038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:36.458890 10038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:36.458963 10038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:36.459012 10038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:36.471232 10038 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:38865
I20260620 19:05:36.471334 10166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:38865 every 8 connection(s)
I20260620 19:05:36.472352 10038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:36.480844 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 10038
I20260620 19:05:36.480973 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:05:36.481225 1642 external_mini_cluster.cc:1428] Setting key 80272ab6803577bd0902ecb3e104ca78
Jun 20 19:05:36 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:36.489742 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:36.474375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:49055 (local address 127.1.154.190:39569)
0620 19:05:36.474520 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:36.474524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:36.476049 (+ 1525us) server_negotiation.cc:407] Connection header received
0620 19:05:36.477437 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:36.477440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:36.477502 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:36.477591 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:36.478823 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:36.479349 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:36.481676 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:36.481976 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:36.482072 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:36.485276 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:36.485280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:36.485283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:36.485309 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:36.487140 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:36.487837 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:36.487841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:36.487842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:36.487920 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:36.488366 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:36.488368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:36.488369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:36.488640 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:36.488750 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:36.489338 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 19:05:36.489490 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:05:36.490468 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:36.474773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:49055)
0620 19:05:36.475828 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:36.475896 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:36.477157 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:36.477795 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:36.477802 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:36.478172 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:36.478683 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:36.478696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:36.479478 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:36.479487 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:36.480472 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:36.480501 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:36.482290 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:36.482293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:36.482501 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:36.483224 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:36.483245 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:36.485106 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:36.487265 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:36.487274 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:36.487293 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:36.487712 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:36.488048 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:36.488054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:36.488059 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:36.488256 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:36.488807 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:36.488812 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:36.489067 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 19:05:36.489240 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 19:05:36.489499 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":116,"threads_started":1}
I20260620 19:05:36.491668 10167 heartbeater.cc:347] Connected to a master server at 127.1.154.190:39569
I20260620 19:05:36.491895 10167 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:36.492375 10167 heartbeater.cc:511] Master 127.1.154.190:39569 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
I20260620 19:05:36.493376 9842 ts_manager.cc:194] Registered new tserver with Master: a4504fdf4a56441e8081a65813719a0f (127.1.154.130:38865)
I20260620 19:05:36.494020 9842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:49055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:05:36.512346 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39569
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:36.633229 10172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:36.633527 10172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:36.633575 10172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:36.638213 10172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:36.638286 10172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:36.638381 10172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:36.645342 10172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:36.646648 10172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:36.647703 10172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:36.658897 10180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:36.658921 10177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:36.658874 10178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:36.659330 10172 server_base.cc:1034] running on GCE node
I20260620 19:05:36.659884 10172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:36.660600 10172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:36.661844 10172 hybrid_clock.cc:648] HybridClock initialized: now 1781982336661806 us; error 56 us; skew 500 ppm
Jun 20 19:05:36 dist-test-slave-6wrf krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:36.665490 10172 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:05:36.667014 10172 webserver.cc:460] Webserver started at http://127.1.154.131:33065/ using document root <none> and password file <none>
I20260620 19:05:36.667759 10172 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:36.667834 10172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:36.668092 10172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:36.670907 10172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "eab345a1e9d84efe9af267ffecfdac38"
format_stamp: "Formatted at 2026-06-20 19:05:36 on dist-test-slave-6wrf"
server_key: "86b399740c2ee7d8d3360c74be3d2f31"
server_key_iv: "9005cc89a81a21d7c0e192f821ae7bdc"
server_key_version: "encryptionkey@0"
I20260620 19:05:36.671577 10172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "eab345a1e9d84efe9af267ffecfdac38"
format_stamp: "Formatted at 2026-06-20 19:05:36 on dist-test-slave-6wrf"
server_key: "86b399740c2ee7d8d3360c74be3d2f31"
server_key_iv: "9005cc89a81a21d7c0e192f821ae7bdc"
server_key_version: "encryptionkey@0"
I20260620 19:05:36.676008 10172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:05:36.679010 10187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:36.680500 10172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:05:36.680612 10172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "eab345a1e9d84efe9af267ffecfdac38"
format_stamp: "Formatted at 2026-06-20 19:05:36 on dist-test-slave-6wrf"
server_key: "86b399740c2ee7d8d3360c74be3d2f31"
server_key_iv: "9005cc89a81a21d7c0e192f821ae7bdc"
server_key_version: "encryptionkey@0"
I20260620 19:05:36.680703 10172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:36.697103 10172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:36.700703 10172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:36.700909 10172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:36.701568 10172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:36.702529 10172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:36.702598 10172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:36.702643 10172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:36.702661 10172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:36.711628 10172 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:34375
I20260620 19:05:36.711694 10300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:34375 every 8 connection(s)
I20260620 19:05:36.712488 10172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:36.720458 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 10172
I20260620 19:05:36.720619 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:05:36.720888 1642 external_mini_cluster.cc:1428] Setting key ac99b35e2604cdf2f91c265e9417051b
Jun 20 19:05:36 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:36.728292 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:36.715168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:56437 (local address 127.1.154.190:39569)
0620 19:05:36.715346 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:36.715350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:36.716489 (+ 1139us) server_negotiation.cc:407] Connection header received
0620 19:05:36.717419 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:36.717422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:36.717480 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:36.717595 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:36.718811 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:36.719352 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:36.720526 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:36.720848 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:36.720978 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:36.724043 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:36.724047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:36.724050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:36.724077 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:36.726318 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:36.726850 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:36.726855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:36.726857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:36.726942 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:36.727263 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:36.727268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:36.727269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:36.727448 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:36.727550 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:36.728013 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 19:05:36.728132 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 19:05:36.729065 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:36.715538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:56437)
0620 19:05:36.716331 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:36.716372 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:36.717209 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:36.717797 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:36.717811 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:36.718182 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:36.718652 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:36.718666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:36.719484 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:36.719488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:36.720397 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:36.720406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:36.721017 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:36.721022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:36.721203 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:36.721898 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:36.721916 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:36.723853 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:36.726477 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:36.726482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:36.726496 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:36.726746 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:36.727039 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:36.727041 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:36.727043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:36.727173 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:36.727545 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:36.727552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:36.727786 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 19:05:36.727972 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 19:05:36.728280 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":115,"threads_started":1}
I20260620 19:05:36.730122 10301 heartbeater.cc:347] Connected to a master server at 127.1.154.190:39569
I20260620 19:05:36.730329 10301 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:36.730768 10301 heartbeater.cc:511] Master 127.1.154.190:39569 requested a full tablet report, sending...
I20260620 19:05:36.731873 9842 ts_manager.cc:194] Registered new tserver with Master: eab345a1e9d84efe9af267ffecfdac38 (127.1.154.131:34375)
I20260620 19:05:36.732435 9842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:56437
I20260620 19:05:36.735985 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.129.keytab.
Entry for principal oryx/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.129.keytab.
WARNING: no policy specified for oryx/127.1.154.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.154.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.130.keytab.
Entry for principal oryx/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.130.keytab.
WARNING: no policy specified for oryx/127.1.154.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.154.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.131.keytab.
Entry for principal oryx/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.131.keytab.
I20260620 19:05:36.825367 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 9904
I20260620 19:05:36.831297 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 10038
I20260620 19:05:36.837491 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 10172
I20260620 19:05:36.842811 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 9810
I20260620 19:05:36.849887 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:39569
--webserver_interface=127.1.154.190
--webserver_port=40175
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:39569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:36.982503 10310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:36.982828 10310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:36.982904 10310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:36.986997 10310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:36.987088 10310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:36.987129 10310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:36.987164 10310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:36.987201 10310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:36.993000 10310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:39569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:39569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=40175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:36.994205 10310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:36.995227 10310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:37.003954 10316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:37.003954 10318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:37.004321 10315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:37.004422 10310 server_base.cc:1034] running on GCE node
I20260620 19:05:37.005018 10310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:37.006273 10310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:37.007499 10310 hybrid_clock.cc:648] HybridClock initialized: now 1781982337007471 us; error 46 us; skew 500 ppm
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982337, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:37.010456 10310 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:05:37.011417 10310 webserver.cc:460] Webserver started at http://127.1.154.190:40175/ using document root <none> and password file <none>
I20260620 19:05:37.012033 10310 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:37.012144 10310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:37.017382 10310 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:37.020340 10325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:37.022097 10310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:37.022271 10310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "376d3c57ac594da78966f8fce8ae8d17"
format_stamp: "Formatted at 2026-06-20 19:05:35 on dist-test-slave-6wrf"
server_key: "46b8d2b868d784fa3d5a62f02157ce25"
server_key_iv: "633278282d5f74d282babbaabe7a2a60"
server_key_version: "encryptionkey@0"
I20260620 19:05:37.022882 10310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:37.060573 10310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:37.064198 10310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:37.064443 10310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:37.071974 10310 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:39569
I20260620 19:05:37.072050 10377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:39569 every 8 connection(s)
I20260620 19:05:37.073156 10310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:37.077436 10378 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:05:37.079041 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 10310
I20260620 19:05:37.079170 10378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17: Bootstrap starting.
I20260620 19:05:37.079567 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:40957
--local_ip_for_outbound_sockets=127.1.154.129
--tserver_master_addrs=127.1.154.190:39569
--webserver_port=45037
--webserver_interface=127.1.154.129
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.154.129
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:05:37.096410 10378 log.cc:824] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:37.103137 10378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:05:37.103559 10378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17: Bootstrap complete.
I20260620 19:05:37.107932 10378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376d3c57ac594da78966f8fce8ae8d17" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39569 } }
I20260620 19:05:37.108381 10378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376d3c57ac594da78966f8fce8ae8d17, State: Initialized, Role: FOLLOWER
I20260620 19:05:37.108922 10378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376d3c57ac594da78966f8fce8ae8d17" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39569 } }
I20260620 19:05:37.109076 10378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:37.109164 10378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:37.109290 10378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:05:37.111673 10378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376d3c57ac594da78966f8fce8ae8d17" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39569 } }
I20260620 19:05:37.112047 10378 leader_election.cc:302] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 376d3c57ac594da78966f8fce8ae8d17; no voters:
I20260620 19:05:37.112385 10378 leader_election.cc:288] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:05:37.112620 10382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:05:37.113085 10382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [term 2 LEADER]: Becoming Leader. State: Replica: 376d3c57ac594da78966f8fce8ae8d17, State: Running, Role: LEADER
I20260620 19:05:37.113662 10382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376d3c57ac594da78966f8fce8ae8d17" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39569 } }
I20260620 19:05:37.113770 10378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:37.115599 10384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 376d3c57ac594da78966f8fce8ae8d17. Latest consensus state: current_term: 2 leader_uuid: "376d3c57ac594da78966f8fce8ae8d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376d3c57ac594da78966f8fce8ae8d17" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39569 } } }
I20260620 19:05:37.116525 10384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:37.116080 10383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "376d3c57ac594da78966f8fce8ae8d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376d3c57ac594da78966f8fce8ae8d17" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39569 } } }
I20260620 19:05:37.116743 10383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:37.119223 10397 catalog_manager.cc:1259] Loaded cluster ID: 00ec15c3b21146e28a8c1deb79047ca5
I20260620 19:05:37.119284 10397 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17: loading cluster ID for follower catalog manager: success
I20260620 19:05:37.121671 10397 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17: acquiring CA information for follower catalog manager: success
I20260620 19:05:37.123163 10397 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:05:37.123291 10399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:37.123963 10399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:37.124344 10399 catalog_manager.cc:1259] Loaded cluster ID: 00ec15c3b21146e28a8c1deb79047ca5
I20260620 19:05:37.124414 10399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:37.125032 10399 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:37.125427 10399 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 376d3c57ac594da78966f8fce8ae8d17: Loaded TSK: 0
I20260620 19:05:37.126286 10399 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:05:37.237017 10380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:37.237382 10380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:37.237449 10380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:37.243620 10380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:37.243713 10380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:37.243803 10380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:05:37.251861 10380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.129.keytab
--principal=oryx/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:40957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=45037
--tserver_master_addrs=127.1.154.190:39569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:37.253185 10380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:37.254472 10380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:37.266813 10406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:37.266812 10405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:37.267243 10408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:37.267283 10380 server_base.cc:1034] running on GCE node
I20260620 19:05:37.267696 10380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:37.268366 10380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:37.269681 10380 hybrid_clock.cc:648] HybridClock initialized: now 1781982337269599 us; error 91 us; skew 500 ppm
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982337, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:37.273759 10380 init.cc:377] Logged in from keytab as oryx/127.1.154.129@KRBTEST.COM (short username oryx)
I20260620 19:05:37.274699 10380 webserver.cc:460] Webserver started at http://127.1.154.129:45037/ using document root <none> and password file <none>
I20260620 19:05:37.275427 10380 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:37.275552 10380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:37.279616 10380 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:05:37.281793 10415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:37.283113 10380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:37.283208 10380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "af7da3feb0bc446db1404085c166e8d1"
format_stamp: "Formatted at 2026-06-20 19:05:36 on dist-test-slave-6wrf"
server_key: "536b8018af687065e95494d327a37bae"
server_key_iv: "c1b2f14edd2b0db3c3682de30f96e4ba"
server_key_version: "encryptionkey@0"
I20260620 19:05:37.283756 10380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:37.319164 10380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:37.323071 10380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:37.323518 10380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:37.324239 10380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:37.325116 10380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:37.325165 10380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:37.325206 10380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:37.325222 10380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:37.338176 10380 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:40957
I20260620 19:05:37.338191 10528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:40957 every 8 connection(s)
I20260620 19:05:37.339332 10380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:05:37.348692 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 10380
I20260620 19:05:37.349351 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:38865
--local_ip_for_outbound_sockets=127.1.154.130
--tserver_master_addrs=127.1.154.190:39569
--webserver_port=44585
--webserver_interface=127.1.154.130
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.154.130
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:37.356545 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:37.342362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:48953)
0620 19:05:37.343473 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:37.343532 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:37.344565 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:37.345929 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:37.345939 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:37.346474 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:37.347119 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.347137 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.348560 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.348565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.350681 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.350706 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.351781 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.351796 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.352075 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.352951 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:37.352979 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:37.355428 (+ 2449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":205,"threads_started":1}
W20260620 19:05:37.356814 10531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:37.342341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:48953 (local address 127.1.154.190:39569)
0620 19:05:37.343045 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:37.343067 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:37.343733 (+ 666us) server_negotiation.cc:407] Connection header received
0620 19:05:37.345078 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:37.345091 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:37.345438 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:37.345842 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:37.347342 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.348328 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.350902 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.351457 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.351604 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.355772 (+ 4168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:48953: BlockingRecv error: recv got EOF from 127.1.154.129:48953 (error 108)
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":180,"threads_started":1}
W20260620 19:05:37.357764 10529 heartbeater.cc:644] Failed to heartbeat to 127.1.154.190:39569 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.154.190:39569: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:37.365988 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:37.358454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:37435)
0620 19:05:37.358597 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:37.358618 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:37.358737 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:37.359141 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:37.359144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:37.359348 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:37.359480 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.359486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.360817 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.360819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.361565 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.361577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.362520 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.362524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.362632 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.363390 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:37.363402 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:37.365865 (+ 2463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 19:05:37.366187 10531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:37.358584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:37435 (local address 127.1.154.190:39569)
0620 19:05:37.358728 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:37.358735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:37.358759 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:05:37.358850 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:37.358856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:37.358934 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:37.359056 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:37.359627 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.360623 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.361817 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.362311 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.362481 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.366064 (+ 3583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:37435: BlockingRecv error: recv got EOF from 127.1.154.129:37435 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:37.374064 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:37.366609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:36355)
0620 19:05:37.366852 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:37.366865 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:37.366940 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:37.367430 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:37.367433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:37.367649 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:37.367781 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.367788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.368986 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.368989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.369892 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.369908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.370722 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.370726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.370835 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.371491 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:37.371510 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:37.373929 (+ 2419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171}
W20260620 19:05:37.374409 10531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:37.366721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:36355 (local address 127.1.154.190:39569)
0620 19:05:37.366841 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:37.366846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:37.366905 (+ 59us) server_negotiation.cc:407] Connection header received
0620 19:05:37.367010 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:37.367013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:37.367070 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:37.367159 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:37.367931 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.368833 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.370044 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.370445 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.370585 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.374247 (+ 3662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:36355: BlockingRecv error: recv got EOF from 127.1.154.129:36355 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 19:05:37.374466 10529 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:05:37.476663 10533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:37.476991 10533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:37.477068 10533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:37.483577 10533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:37.483675 10533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:37.483814 10533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:37.490783 10533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.130.keytab
--principal=oryx/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:38865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=44585
--tserver_master_addrs=127.1.154.190:39569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:37.492169 10533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:37.493496 10533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:37.503989 10539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:37.504002 10538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:37.504069 10533 server_base.cc:1034] running on GCE node
W20260620 19:05:37.504002 10541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:37.504621 10533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:37.505275 10533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:37.506485 10533 hybrid_clock.cc:648] HybridClock initialized: now 1781982337506468 us; error 34 us; skew 500 ppm
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982337, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:37.509486 10533 init.cc:377] Logged in from keytab as oryx/127.1.154.130@KRBTEST.COM (short username oryx)
I20260620 19:05:37.510547 10533 webserver.cc:460] Webserver started at http://127.1.154.130:44585/ using document root <none> and password file <none>
I20260620 19:05:37.511232 10533 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:37.511379 10533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:37.515730 10533 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:05:37.518303 10548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:37.519903 10533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:37.520059 10533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "a4504fdf4a56441e8081a65813719a0f"
format_stamp: "Formatted at 2026-06-20 19:05:36 on dist-test-slave-6wrf"
server_key: "aa0d009caa1f5d972328c699cb2ee052"
server_key_iv: "a32310d509d7ae44de1518990789100f"
server_key_version: "encryptionkey@0"
I20260620 19:05:37.520628 10533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:37.545393 10533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:37.548796 10533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:37.548990 10533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:37.549567 10533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:37.550428 10533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:37.550499 10533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:37.550568 10533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:37.550618 10533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:37.560559 10533 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:38865
I20260620 19:05:37.560580 10661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:38865 every 8 connection(s)
I20260620 19:05:37.561556 10533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:37.568626 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 10533
I20260620 19:05:37.569089 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:34375
--local_ip_for_outbound_sockets=127.1.154.131
--tserver_master_addrs=127.1.154.190:39569
--webserver_port=33065
--webserver_interface=127.1.154.131
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.154.131
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:05:37.573300 10531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:37.563943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:57915 (local address 127.1.154.190:39569)
0620 19:05:37.564117 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:37.564121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:37.565500 (+ 1379us) server_negotiation.cc:407] Connection header received
0620 19:05:37.566412 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:37.566415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:37.566471 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:37.566583 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:37.567718 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.568238 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.569517 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.569806 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.569921 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.573182 (+ 3261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:57915: BlockingRecv error: recv got EOF from 127.1.154.130:57915 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 19:05:37.573982 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:37.564466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:57915)
0620 19:05:37.565338 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:37.565380 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:37.566209 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:37.566756 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:37.566762 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:37.567116 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:37.567573 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.567586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.568383 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.568386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.569400 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.569410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.569959 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.569963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.570128 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.570732 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:37.570748 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:37.573069 (+ 2321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":179,"threads_started":1}
W20260620 19:05:37.574429 10662 heartbeater.cc:644] Failed to heartbeat to 127.1.154.190:39569 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.154.190:39569: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:37.581076 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:37.574915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:45273)
0620 19:05:37.575061 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:37.575077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:37.575163 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:37.575469 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:37.575472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:37.575617 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:37.575706 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.575711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.576945 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.576950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.577711 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.577726 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.578489 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.578493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.578583 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.579168 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:37.579188 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:37.580973 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 19:05:37.581245 10531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:37.575030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:45273 (local address 127.1.154.190:39569)
0620 19:05:37.575191 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:37.575196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:37.575215 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:05:37.575260 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:37.575265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:37.575319 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:37.575432 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:37.575823 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.576696 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.577880 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.578317 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.578471 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.581130 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:45273: BlockingRecv error: recv got EOF from 127.1.154.130:45273 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:37.587401 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:37.581855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:50137)
0620 19:05:37.582015 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:37.582027 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:37.582100 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:37.582334 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:37.582336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:37.582457 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:37.582531 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.582536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.583536 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.583539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.584077 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.584085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.584625 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.584628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.584687 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.585183 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:37.585195 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:37.587275 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 19:05:37.587554 10531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:37.581923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:50137 (local address 127.1.154.190:39569)
0620 19:05:37.582031 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:37.582035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:37.582067 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:05:37.582183 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:37.582186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:37.582231 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:37.582312 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:37.582626 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.583405 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.584178 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.584516 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.584613 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.587441 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:50137: BlockingRecv error: recv got EOF from 127.1.154.130:50137 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 19:05:37.587852 10662 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:05:37.697078 10665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:37.697357 10665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:37.697407 10665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:37.701889 10665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:37.701948 10665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:37.702008 10665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:37.707237 10665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/krb5kdc/oryx_127.1.154.131.keytab
--principal=oryx/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:34375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=33065
--tserver_master_addrs=127.1.154.190:39569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:37.708210 10665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:37.709144 10665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:37.721212 10670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:37.721453 10665 server_base.cc:1034] running on GCE node
W20260620 19:05:37.721225 10671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:37.721235 10673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:37.722054 10665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:37.722681 10665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:37.723855 10665 hybrid_clock.cc:648] HybridClock initialized: now 1781982337723837 us; error 29 us; skew 500 ppm
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982337, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:37.732744 10665 init.cc:377] Logged in from keytab as oryx/127.1.154.131@KRBTEST.COM (short username oryx)
I20260620 19:05:37.734411 10665 webserver.cc:460] Webserver started at http://127.1.154.131:33065/ using document root <none> and password file <none>
I20260620 19:05:37.735334 10665 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:37.735459 10665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:37.740059 10665 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:05:37.742650 10680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:37.744202 10665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:37.744367 10665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "eab345a1e9d84efe9af267ffecfdac38"
format_stamp: "Formatted at 2026-06-20 19:05:36 on dist-test-slave-6wrf"
server_key: "86b399740c2ee7d8d3360c74be3d2f31"
server_key_iv: "9005cc89a81a21d7c0e192f821ae7bdc"
server_key_version: "encryptionkey@0"
I20260620 19:05:37.744949 10665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:37.760062 10665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:37.763948 10665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:37.764184 10665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:37.764827 10665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:37.765843 10665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:37.765919 10665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:37.766002 10665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:37.766057 10665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:37.777086 10665 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:34375
I20260620 19:05:37.777187 10793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:34375 every 8 connection(s)
I20260620 19:05:37.778484 10665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:37.787806 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 10665
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:05:37.792279 10531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:37.780956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:39207 (local address 127.1.154.190:39569)
0620 19:05:37.781095 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:37.781098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:37.782788 (+ 1690us) server_negotiation.cc:407] Connection header received
0620 19:05:37.783719 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:37.783722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:37.783778 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:37.783888 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:37.785073 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.785652 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.787326 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.787620 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.787704 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.792062 (+ 4358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:39207: BlockingRecv error: recv got EOF from 127.1.154.131:39207 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:05:37.792858 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:37.781702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:39207)
0620 19:05:37.782583 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:37.782631 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:37.783505 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:37.784069 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:37.784076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:37.784440 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:37.784935 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.784949 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.785906 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.785912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.787041 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.787054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.787792 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.787797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.788000 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.788726 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:37.788742 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:37.791777 (+ 3035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":203,"threads_started":1}
W20260620 19:05:37.793418 10794 heartbeater.cc:644] Failed to heartbeat to 127.1.154.190:39569 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.154.190:39569: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:37.799829 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:37.788977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.1.154.190:39569)
0620 19:05:37.789196 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:37.789199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:37.789211 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:05:37.789239 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:37.789247 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:37.789280 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:37.789354 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:37.790385 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.791061 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.792300 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.792695 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.792808 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.794980 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:37.794994 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:37.795008 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:37.795037 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:37.797003 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:37.797951 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:37.797956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:37.797961 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:37.798091 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:37.798474 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:37.798478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:37.798479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:37.798881 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:37.799098 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:37.799372 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 19:05:37.799633 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":93,"threads_started":1}
W20260620 19:05:37.800226 10531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:37.794094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:59003 (local address 127.1.154.190:39569)
0620 19:05:37.794237 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:37.794243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:37.794282 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:05:37.794403 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:37.794408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:37.794462 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:37.794553 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:37.794916 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.795402 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.796280 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.796492 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.796594 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.800137 (+ 3543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:59003: BlockingRecv error: recv got EOF from 127.1.154.131:59003 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:05:37.800475 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:37.793951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:59003)
0620 19:05:37.794218 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:37.794238 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:37.794314 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:37.794576 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:37.794577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:37.794704 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:37.794800 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.794805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.795527 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.795530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.796155 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.796163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.796595 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.796598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.796688 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.797326 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:37.797347 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:37.800356 (+ 3009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201}
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:37 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:37.806612 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:37.801236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:38611)
0620 19:05:37.801391 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:37.801403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:37.801468 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:37.801892 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:37.801895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:37.802050 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:37.802154 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.802160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.803045 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.803048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.803576 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:37.803584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.804045 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.804048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:37.804112 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.804622 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:37.804635 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:37.806519 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260620 19:05:37.806794 10798 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:37.801277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:38611 (local address 127.1.154.190:39569)
0620 19:05:37.801629 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:37.801634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:37.801647 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:05:37.801681 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:37.801687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:37.801732 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:37.801795 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:37.802286 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.802923 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.803680 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:37.803898 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:37.803978 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:37.806659 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:38611: BlockingRecv error: recv got EOF from 127.1.154.131:38611 (error 108)
Metrics: {"server-negotiator.queue_time_us":250}
W20260620 19:05:37.806857 10794 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 19:05:38 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:38 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:38.382095 10799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:38.375243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:57439)
0620 19:05:38.375556 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:38.375573 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:38.375681 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:38.375976 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:38.375978 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:38.376151 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:38.376266 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:38.376271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:38.377042 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:38.377045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:38.377660 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:38.377669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:38.378633 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:38.378638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:38.378745 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:38.379612 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:38.379626 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:38.381915 (+ 2289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":110,"threads_started":1}
W20260620 19:05:38.382181 10800 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:38.375294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:57439 (local address 127.1.154.190:39569)
0620 19:05:38.375595 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:38.375602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:38.375622 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:05:38.375820 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:38.375823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:38.375872 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:38.375955 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:38.376377 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:38.376909 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:38.377892 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:38.378384 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:38.378539 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:38.382055 (+ 3516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:57439: BlockingRecv error: recv got EOF from 127.1.154.129:57439 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":108,"threads_started":1}
Jun 20 19:05:38 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:38 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:38.595185 10801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:38.588583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:39339)
0620 19:05:38.588835 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:38.588854 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:38.588958 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:38.589268 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:38.589271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:38.589481 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:38.589565 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:38.589571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:38.590454 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:38.590458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:38.591192 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:38.591202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:38.591707 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:38.591710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:38.591773 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:38.592538 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:38.592550 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:38.595058 (+ 2508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":96,"threads_started":1}
W20260620 19:05:38.595736 10800 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:38.588612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:39339 (local address 127.1.154.190:39569)
0620 19:05:38.588812 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:38.588817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:38.588922 (+ 105us) server_negotiation.cc:407] Connection header received
0620 19:05:38.589089 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:38.589093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:38.589152 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:38.589248 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:38.589775 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:38.590300 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:38.591362 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:38.591577 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:38.591664 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:38.595584 (+ 3920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:39339: BlockingRecv error: recv got EOF from 127.1.154.130:39339 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 19:05:38 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:38 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:38.814031 10802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:38.807547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:50709)
0620 19:05:38.807909 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:38.807927 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:38.808075 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:38.808347 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:38.808350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:38.808551 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:38.808637 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:38.808642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:38.809441 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:38.809447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:38.810078 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:38.810086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:38.810526 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:38.810529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:38.810610 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:38.811147 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:38.811165 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:38.813754 (+ 2589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":122,"threads_started":1}
W20260620 19:05:38.814086 10800 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:38.807747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:50709 (local address 127.1.154.190:39569)
0620 19:05:38.807910 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:38.807916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:38.808012 (+ 96us) server_negotiation.cc:407] Connection header received
0620 19:05:38.808183 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:38.808192 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:38.808238 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:38.808326 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:38.808753 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:38.809273 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:38.810190 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:38.810413 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:38.810523 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:38.813956 (+ 3433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:50709: BlockingRecv error: recv got EOF from 127.1.154.131:50709 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 19:05:39 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:39 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:39.389847 10803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:39.383249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:52953)
0620 19:05:39.383552 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:39.383567 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:39.383662 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:39.384331 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:39.384334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:39.384528 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:39.384624 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:39.384630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:39.385510 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:39.385514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:39.386300 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:39.386312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:39.386961 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:39.386966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:39.387068 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:39.387712 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:39.387728 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:39.389702 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":114,"threads_started":1}
W20260620 19:05:39.390151 10804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:39.383375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:52953 (local address 127.1.154.190:39569)
0620 19:05:39.383700 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:39.383704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:39.383722 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:05:39.383778 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:39.383780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:39.383832 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:39.384121 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:39.384771 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:39.385330 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:39.386478 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:39.386816 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:39.386924 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:39.389872 (+ 2948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:52953: BlockingRecv error: recv got EOF from 127.1.154.129:52953 (error 108)
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":124,"threads_started":1}
Jun 20 19:05:39 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:39 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:39.603276 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:39.596426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:35317)
0620 19:05:39.596776 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:39.596801 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:39.596938 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:39.597413 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:39.597417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:39.597675 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:39.597809 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:39.597816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:39.598667 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:39.598670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:39.599300 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:39.599313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:39.600009 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:39.600014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:39.600164 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:39.601032 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:39.601046 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:39.603135 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":127,"threads_started":1}
W20260620 19:05:39.603703 10804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:39.596603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:35317 (local address 127.1.154.190:39569)
0620 19:05:39.596816 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:39.596820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:39.596880 (+ 60us) server_negotiation.cc:407] Connection header received
0620 19:05:39.597144 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:39.597148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:39.597213 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:39.597303 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:39.598016 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:39.598548 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:39.599452 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:39.599765 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:39.599884 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:39.603536 (+ 3652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:35317: BlockingRecv error: recv got EOF from 127.1.154.130:35317 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 20 19:05:39 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:39 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:05:39.822151 10804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:39.815190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:42069 (local address 127.1.154.190:39569)
0620 19:05:39.815328 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:39.815331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:39.815347 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:05:39.815467 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:39.815469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:39.815516 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:39.815647 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:39.816064 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:39.816698 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:39.818196 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:39.818419 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:39.818511 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:39.822047 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:42069: BlockingRecv error: recv got EOF from 127.1.154.131:42069 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:05:39.822142 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:39.814991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:42069)
0620 19:05:39.815265 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:39.815280 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:39.815380 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:39.815664 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:39.815666 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:39.815842 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:39.815937 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:39.815942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:39.816908 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:39.816914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:39.818053 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:39.818068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:39.818563 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:39.818567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:39.818663 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:39.819244 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:39.819256 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:39.821913 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":103,"threads_started":1}
Jun 20 19:05:40 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:40 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:40.397637 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:40.390700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:49279)
0620 19:05:40.390992 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:40.391006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:40.391110 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:40.391531 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:40.391535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:40.391772 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:40.391937 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:40.391945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:40.392985 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:40.392999 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:40.393896 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:40.393908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:40.394402 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:40.394406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:40.394490 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:40.395141 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:40.395156 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:40.397421 (+ 2265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":89,"threads_started":1}
W20260620 19:05:40.397712 10808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:40.390847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:49279 (local address 127.1.154.190:39569)
0620 19:05:40.391062 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:40.391066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:40.391079 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:05:40.391201 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:40.391204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:40.391266 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:40.391354 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:40.392118 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:40.392745 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:40.394041 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:40.394275 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:40.394386 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:40.397548 (+ 3162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:49279: BlockingRecv error: recv got EOF from 127.1.154.129:49279 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
Jun 20 19:05:40 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:40 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:40.610818 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:40.604196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:43843)
0620 19:05:40.604470 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:40.604484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:40.604572 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:40.604989 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:40.604996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:40.605259 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:40.605365 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:40.605371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:40.606376 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:40.606381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:40.607090 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:40.607104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:40.607851 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:40.607855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:40.607932 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:40.608578 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:40.608590 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:40.610688 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
W20260620 19:05:40.611259 10808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:40.604464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:43843 (local address 127.1.154.190:39569)
0620 19:05:40.604634 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:40.604639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:40.604656 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:05:40.604704 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:40.604707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:40.604760 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:40.604862 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:40.605570 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:40.606161 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:40.607270 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:40.607525 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:40.607620 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:40.611084 (+ 3464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:43843: BlockingRecv error: recv got EOF from 127.1.154.130:43843 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 19:05:40 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:40 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:40.830425 10810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:40.822976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:44663)
0620 19:05:40.823358 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:40.823379 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:40.823555 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:40.824015 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:40.824021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:40.824328 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:40.824524 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:40.824531 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:40.825501 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:40.825505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:40.826485 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:40.826498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:40.827418 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:40.827440 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:40.827555 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:40.828233 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:40.828246 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:40.830286 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":90,"threads_started":1}
W20260620 19:05:40.830799 10808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:40.823206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:44663 (local address 127.1.154.190:39569)
0620 19:05:40.823425 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:40.823429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:40.823483 (+ 54us) server_negotiation.cc:407] Connection header received
0620 19:05:40.823696 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:40.823699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:40.823761 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:40.823858 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:40.824734 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:40.825291 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:40.826658 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:40.827197 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:40.827312 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:40.830640 (+ 3328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:44663: BlockingRecv error: recv got EOF from 127.1.154.131:44663 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Jun 20 19:05:41 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:41 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:41.410598 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:41.399077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:53267)
0620 19:05:41.399414 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:41.399428 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:41.399545 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:41.400267 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:41.400269 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:41.400445 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:41.400564 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:41.400569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:41.401753 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:41.401755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:41.402439 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:41.402448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:41.403067 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:41.403071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:41.403194 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:41.407778 (+ 4584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:41.407806 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:41.410437 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":100,"threads_started":1}
W20260620 19:05:41.410928 10811 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:41.398967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:53267 (local address 127.1.154.190:39569)
0620 19:05:41.399302 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:41.399307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:41.399894 (+ 587us) server_negotiation.cc:407] Connection header received
0620 19:05:41.399979 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:41.399982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:41.400054 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:41.400188 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:41.400746 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:41.401500 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:41.402574 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:41.402878 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:41.402968 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:41.410738 (+ 7770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:53267: BlockingRecv error: recv got EOF from 127.1.154.129:53267 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":83,"threads_started":1}
Jun 20 19:05:41 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:41 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:41.620580 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:41.611902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:35583)
0620 19:05:41.612209 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:41.612228 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:41.612353 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:41.613118 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:41.613122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:41.613440 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:41.613588 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:41.613651 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:41.614956 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:41.614960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:41.615633 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:41.615643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:41.616335 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:41.616338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:41.616421 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:41.617056 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:41.617069 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:41.620352 (+ 3283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":143,"threads_started":1}
W20260620 19:05:41.620710 10811 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:41.612187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:35583 (local address 127.1.154.190:39569)
0620 19:05:41.612397 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:41.612404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:41.612425 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:05:41.612614 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:41.612618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:41.612740 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:41.612903 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:41.613801 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:41.614756 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:41.615776 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:41.616184 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:41.616332 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:41.620570 (+ 4238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:35583: BlockingRecv error: recv got EOF from 127.1.154.130:35583 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 19:05:41 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:41 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:41.840257 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:41.832094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:43591)
0620 19:05:41.832678 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:41.832697 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:41.832855 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:41.833268 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:41.833271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:41.833643 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:41.833976 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:41.833990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:41.835581 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:41.835583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:41.836147 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:41.836154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:41.836666 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:41.836669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:41.836734 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:41.837369 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:41.837381 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:41.840038 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":136,"threads_started":1}
W20260620 19:05:41.840345 10811 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:41.832093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:43591 (local address 127.1.154.190:39569)
0620 19:05:41.832444 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:41.832450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:41.832843 (+ 393us) server_negotiation.cc:407] Connection header received
0620 19:05:41.832953 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:41.832956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:41.833017 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:41.833124 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:41.834192 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:41.835392 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:41.836274 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:41.836489 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:41.836582 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:41.840225 (+ 3643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:43591: BlockingRecv error: recv got EOF from 127.1.154.131:43591 (error 108)
Metrics: {"server-negotiator.queue_time_us":225}
Jun 20 19:05:42 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:42 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:42.419960 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:42.411652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:41167)
0620 19:05:42.411938 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:42.411951 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:42.412042 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:42.413085 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:42.413088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:42.413479 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:42.413792 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:42.413814 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:42.415078 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:42.415081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:42.416048 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:42.416060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:42.416844 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:42.416848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:42.416942 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:42.417764 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:42.417777 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:42.419826 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":110,"threads_started":1}
W20260620 19:05:42.420228 10816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:42.411905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:41167 (local address 127.1.154.190:39569)
0620 19:05:42.412629 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:42.412633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:42.412652 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:05:42.412779 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:42.412785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:42.412857 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:42.412996 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:42.413963 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:42.414920 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:42.416230 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:42.416659 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:42.416781 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:42.420061 (+ 3280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:41167: BlockingRecv error: recv got EOF from 127.1.154.129:41167 (error 108)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":256,"threads_started":1}
Jun 20 19:05:42 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:42 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:42.630285 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:42.621997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:55289)
0620 19:05:42.622418 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:42.622436 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:42.622565 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:42.622957 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:42.622961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:42.623379 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:42.623566 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:42.623576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:42.624894 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:42.624900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:42.626031 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:42.626044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:42.626686 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:42.626692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:42.626833 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:42.627514 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:42.627538 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:42.630145 (+ 2607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":221,"threads_started":1}
W20260620 19:05:42.630486 10816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:42.622205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:55289 (local address 127.1.154.190:39569)
0620 19:05:42.622346 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:42.622349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:42.622537 (+ 188us) server_negotiation.cc:407] Connection header received
0620 19:05:42.622697 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:42.622700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:42.622747 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:42.622843 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:42.623721 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:42.624533 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:42.626161 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:42.626480 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:42.626634 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:42.630314 (+ 3680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:55289: BlockingRecv error: recv got EOF from 127.1.154.130:55289 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 19:05:42 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:42 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:42.848556 10818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:42.841723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:40407)
0620 19:05:42.842050 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:42.842064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:42.842169 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:42.842501 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:42.842504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:42.842699 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:42.842795 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:42.842800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:42.843625 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:42.843627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:42.844181 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:42.844189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:42.844930 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:42.844937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:42.845055 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:42.845886 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:42.845904 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:42.848344 (+ 2440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":100,"threads_started":1}
W20260620 19:05:42.848786 10816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:42.841749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:40407 (local address 127.1.154.190:39569)
0620 19:05:42.841910 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:42.841915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:42.842182 (+ 267us) server_negotiation.cc:407] Connection header received
0620 19:05:42.842273 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:42.842275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:42.842341 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:42.842422 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:42.842973 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:42.843500 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:42.844365 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:42.844727 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:42.844892 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:42.848589 (+ 3697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:40407: BlockingRecv error: recv got EOF from 127.1.154.131:40407 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 19:05:43 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:43 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:43.427632 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:43.420842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:35231)
0620 19:05:43.421093 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:43.421134 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:43.421234 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:43.421892 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:43.421895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:43.422129 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:43.422258 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:43.422265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:43.423170 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:43.423172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:43.423869 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:43.423879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:43.424417 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:43.424420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:43.424484 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:43.425183 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:43.425195 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:43.427512 (+ 2317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":94,"threads_started":1}
W20260620 19:05:43.427840 10820 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:43.420984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:35231 (local address 127.1.154.190:39569)
0620 19:05:43.421387 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:43.421391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:43.421411 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:05:43.421468 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:43.421471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:43.421526 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:43.421768 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:43.422487 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:43.423031 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:43.423999 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:43.424279 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:43.424382 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:43.427688 (+ 3306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:35231: BlockingRecv error: recv got EOF from 127.1.154.129:35231 (error 108)
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":196,"threads_started":1}
Jun 20 19:05:43 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:43 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:43.637279 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:43.631221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:40647)
0620 19:05:43.631512 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:43.631554 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:43.631671 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:43.631979 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:43.631982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:43.632208 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:43.632302 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:43.632308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:43.633099 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:43.633105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:43.633750 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:43.633759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:43.634229 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:43.634233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:43.634312 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:43.635114 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:43.635128 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:43.637100 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":117,"threads_started":1}
W20260620 19:05:43.637337 10820 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:43.631479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:40647 (local address 127.1.154.190:39569)
0620 19:05:43.631661 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:43.631666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:43.631684 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:05:43.631776 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:43.631780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:43.631837 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:43.631958 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:43.632446 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:43.632980 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:43.633884 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:43.634108 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:43.634225 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:43.637229 (+ 3004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:40647: BlockingRecv error: recv got EOF from 127.1.154.130:40647 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 19:05:43 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:43 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:43.855516 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:43.849491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:34491)
0620 19:05:43.849779 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:43.849800 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:43.849947 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:43.850314 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:43.850319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:43.850558 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:43.850648 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:43.850654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:43.851403 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:43.851406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:43.851969 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:43.851976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:43.852399 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:43.852411 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:43.852482 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:43.853058 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:43.853070 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:43.855396 (+ 2326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":128,"threads_started":1}
W20260620 19:05:43.855866 10820 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:43.849515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:34491 (local address 127.1.154.190:39569)
0620 19:05:43.849686 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:43.849691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:43.849904 (+ 213us) server_negotiation.cc:407] Connection header received
0620 19:05:43.850089 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:43.850092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:43.850143 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:43.850252 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:43.850754 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:43.851301 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:43.852074 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:43.852278 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:43.852357 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:43.855742 (+ 3385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:34491: BlockingRecv error: recv got EOF from 127.1.154.131:34491 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 19:05:44 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:44 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:44.434315 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:44.428480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:36027)
0620 19:05:44.428856 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:44.428872 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:44.428989 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:44.429343 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:44.429345 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:44.429555 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:44.429679 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:44.429685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:44.430599 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:44.430604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:44.431153 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:44.431162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:44.431609 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:44.431612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:44.431675 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:44.432182 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:44.432193 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:44.434199 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":90,"threads_started":1}
W20260620 19:05:44.434463 10824 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:44.428625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:36027 (local address 127.1.154.190:39569)
0620 19:05:44.428864 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:44.428868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:44.429019 (+ 151us) server_negotiation.cc:407] Connection header received
0620 19:05:44.429099 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:44.429102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:44.429180 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:44.429301 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:44.429815 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:44.430348 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:44.431274 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:44.431488 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:44.431573 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:44.434331 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:36027: BlockingRecv error: recv got EOF from 127.1.154.129:36027 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":63,"threads_started":1}
Jun 20 19:05:44 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:44 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:44.645623 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:44.638449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:33253)
0620 19:05:44.638777 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:44.638797 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:44.638919 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:44.639339 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:44.639342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:44.639593 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:44.639788 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:44.639802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:44.640770 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:44.640773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:44.641396 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:44.641404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:44.642025 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:44.642029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:44.642124 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:44.642999 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:44.643016 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:44.645417 (+ 2401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":102,"threads_started":1}
W20260620 19:05:44.645709 10824 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:44.638521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:33253 (local address 127.1.154.190:39569)
0620 19:05:44.638660 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:44.638664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:44.638918 (+ 254us) server_negotiation.cc:407] Connection header received
0620 19:05:44.639050 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:44.639053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:44.639121 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:44.639219 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:44.639968 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:44.640602 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:44.641556 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:44.641845 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:44.641951 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:44.645565 (+ 3614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:33253: BlockingRecv error: recv got EOF from 127.1.154.130:33253 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 19:05:44 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:44 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:44.863760 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:44.856515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:47753)
0620 19:05:44.856832 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:44.856846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:44.856936 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:44.857429 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:44.857433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:44.857991 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:44.858115 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:44.858123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:44.859094 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:44.859097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:44.859852 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:44.859863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:44.860422 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:44.860426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:44.860501 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:44.861236 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:44.861249 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:44.863632 (+ 2383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":92,"threads_started":1}
W20260620 19:05:44.864025 10824 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:44.856675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:47753 (local address 127.1.154.190:39569)
0620 19:05:44.856912 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:44.856916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:44.856937 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:05:44.857077 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:44.857080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:44.857143 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:44.857262 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:44.858297 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:44.858889 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:44.860006 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:44.860248 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:44.860350 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:44.863870 (+ 3520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:47753: BlockingRecv error: recv got EOF from 127.1.154.131:47753 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Jun 20 19:05:45 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:45 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:45.441468 10827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:45.435450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:40665)
0620 19:05:45.435762 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:45.435777 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:45.435882 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:45.436160 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:45.436169 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:45.436417 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:45.436529 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:45.436535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:45.437391 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:45.437394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:45.438019 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:45.438027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:45.438511 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:45.438514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:45.438576 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:45.439125 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:45.439137 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:45.441347 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":94,"threads_started":1}
W20260620 19:05:45.441715 10828 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:45.435654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:40665 (local address 127.1.154.190:39569)
0620 19:05:45.435918 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:45.435922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:45.435942 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:05:45.435992 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:45.435994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:45.436035 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:45.436107 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:45.436666 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:45.437258 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:45.438147 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:45.438383 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:45.438475 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:45.441512 (+ 3037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:40665: BlockingRecv error: recv got EOF from 127.1.154.129:40665 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":103,"threads_started":1}
Jun 20 19:05:45 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:45 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:45.653133 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:45.646542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:60887)
0620 19:05:45.646918 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:45.646930 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:45.647069 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:45.647343 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:45.647346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:45.647576 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:45.647668 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:45.647674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:45.648665 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:45.648670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:45.649265 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:45.649274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:45.649865 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:45.649868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:45.649951 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:45.650887 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:45.650902 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:45.652956 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":112,"threads_started":1}
W20260620 19:05:45.653167 10828 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:45.646668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:60887 (local address 127.1.154.190:39569)
0620 19:05:45.646855 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:45.646860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:45.647007 (+ 147us) server_negotiation.cc:407] Connection header received
0620 19:05:45.647182 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:45.647185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:45.647238 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:45.647331 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:45.647783 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:45.648559 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:45.649392 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:45.649747 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:45.649858 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:45.653070 (+ 3212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:60887: BlockingRecv error: recv got EOF from 127.1.154.130:60887 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 19:05:45 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:45 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:45.870646 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:45.864835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:54455)
0620 19:05:45.865212 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:45.865233 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:45.865397 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:45.865711 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:45.865714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:45.865931 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:45.866038 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:45.866044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:45.866781 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:45.866786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:45.867343 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:45.867350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:45.867780 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:45.867783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:45.867872 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:45.868428 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:45.868443 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:45.870521 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":148,"threads_started":1}
W20260620 19:05:45.870872 10828 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:45.865005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:54455 (local address 127.1.154.190:39569)
0620 19:05:45.865163 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:45.865168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:45.865331 (+ 163us) server_negotiation.cc:407] Connection header received
0620 19:05:45.865501 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:45.865505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:45.865558 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:45.865688 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:45.866151 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:45.866673 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:45.867457 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:45.867668 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:45.867776 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:45.870729 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:54455: BlockingRecv error: recv got EOF from 127.1.154.131:54455 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 19:05:46 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:46 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:46.453817 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:46.442614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:55207)
0620 19:05:46.443035 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:46.443059 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:46.443216 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:46.443528 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:46.443530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:46.443781 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:46.443895 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:46.443900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:46.445084 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:46.445090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:46.445824 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:46.445837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:46.446596 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:46.446599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:46.446661 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:46.450869 (+ 4208us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:46.450887 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:46.453558 (+ 2671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":98,"threads_started":1}
W20260620 19:05:46.454017 10832 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:46.442757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:55207 (local address 127.1.154.190:39569)
0620 19:05:46.443182 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:46.443186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:46.443204 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:05:46.443311 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:46.443313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:46.443361 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:46.443467 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:46.444057 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:46.444894 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:46.446067 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:46.446471 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:46.446563 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:46.453808 (+ 7245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:55207: BlockingRecv error: recv got EOF from 127.1.154.129:55207 (error 108)
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":180,"threads_started":1}
Jun 20 19:05:46 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:46 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:46.661681 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:46.654361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:45295)
0620 19:05:46.654630 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:46.654643 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:46.654788 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:46.655056 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:46.655059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:46.655273 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:46.655356 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:46.655362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:46.656559 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:46.656566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:46.657192 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:46.657202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:46.657815 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:46.657818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:46.657909 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:46.658729 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:46.658743 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:46.661452 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
W20260620 19:05:46.662034 10832 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:46.654419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:45295 (local address 127.1.154.190:39569)
0620 19:05:46.654566 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:46.654571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:46.654727 (+ 156us) server_negotiation.cc:407] Connection header received
0620 19:05:46.654888 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:46.654892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:46.654948 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:46.655036 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:46.655468 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:46.656251 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:46.657368 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:46.657684 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:46.657806 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:46.661757 (+ 3951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:45295: BlockingRecv error: recv got EOF from 127.1.154.130:45295 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 19:05:46 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:46 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:46.878067 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:46.871635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:54341)
0620 19:05:46.872113 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:46.872137 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:46.872404 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:46.872879 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:46.872882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:46.873310 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:46.873422 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:46.873429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:46.874301 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:46.874307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:46.874856 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:46.874863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:46.875328 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:46.875332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:46.875419 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:46.875980 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:46.875993 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:46.877926 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":120,"threads_started":1}
W20260620 19:05:46.878263 10832 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:46.871841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:54341 (local address 127.1.154.190:39569)
0620 19:05:46.872057 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:46.872062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:46.872355 (+ 293us) server_negotiation.cc:407] Connection header received
0620 19:05:46.872648 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:46.872652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:46.872725 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:46.872845 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:46.873583 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:46.874179 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:46.874985 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:46.875214 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:46.875325 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:46.878078 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:54341: BlockingRecv error: recv got EOF from 127.1.154.131:54341 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 20 19:05:47 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:47 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:47.462730 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:47.455065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:51913)
0620 19:05:47.455389 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:47.455404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:47.455503 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:47.455738 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:47.455740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:47.455925 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:47.456043 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:47.456048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:47.456930 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:47.456934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:47.457768 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:47.457783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:47.458593 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:47.458596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:47.458696 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:47.459458 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:47.459478 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:47.462473 (+ 2995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":76,"threads_started":1}
W20260620 19:05:47.463335 10836 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:47.455174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:51913 (local address 127.1.154.190:39569)
0620 19:05:47.455430 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:47.455433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:47.455450 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:05:47.455587 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:47.455589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:47.455631 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:47.455700 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:47.456179 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:47.456739 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:47.457993 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:47.458268 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:47.458532 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:47.462790 (+ 4258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:51913: BlockingRecv error: recv got EOF from 127.1.154.129:51913 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
Jun 20 19:05:47 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:47 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:47.669587 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:47.662573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:57639)
0620 19:05:47.663000 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:47.663017 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:47.663128 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:47.663915 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:47.663920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:47.664148 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:47.664291 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:47.664299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:47.665114 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:47.665116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:47.665749 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:47.665757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:47.666258 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:47.666262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:47.666378 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:47.667279 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:47.667293 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:47.669452 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":98,"threads_started":1}
W20260620 19:05:47.669725 10836 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:47.662732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:57639 (local address 127.1.154.190:39569)
0620 19:05:47.662926 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:47.662932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:47.663150 (+ 218us) server_negotiation.cc:407] Connection header received
0620 19:05:47.663307 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:47.663313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:47.663558 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:47.663669 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:47.664434 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:47.664994 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:47.665854 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:47.666082 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:47.666165 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:47.669626 (+ 3461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:57639: BlockingRecv error: recv got EOF from 127.1.154.130:57639 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 19:05:47 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:47 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:47.885356 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:47.879162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:36555)
0620 19:05:47.879454 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:47.879468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:47.879560 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:47.879828 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:47.879830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:47.880003 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:47.880092 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:47.880097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:47.881187 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:47.881190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:47.881877 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:47.881886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:47.882474 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:47.882477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:47.882555 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:47.883120 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:47.883138 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:47.885242 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":109,"threads_started":1}
W20260620 19:05:47.885555 10836 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:47.879347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:36555 (local address 127.1.154.190:39569)
0620 19:05:47.879521 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:47.879525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:47.879541 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:05:47.879663 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:47.879665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:47.879710 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:47.879793 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:47.880283 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:47.880989 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:47.882058 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:47.882308 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:47.882402 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:47.885417 (+ 3015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:36555: BlockingRecv error: recv got EOF from 127.1.154.131:36555 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 19:05:48 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:48 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:48.476205 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:48.468044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:46179)
0620 19:05:48.468334 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:48.468352 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:48.468498 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:48.468807 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:48.468810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:48.469181 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:48.469295 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:48.469305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:48.470469 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:48.470473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:48.471362 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:48.471373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:48.472017 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:48.472021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:48.472092 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:48.472731 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:48.472749 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:48.476036 (+ 3287us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
W20260620 19:05:48.476307 10840 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:48.468100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:46179 (local address 127.1.154.190:39569)
0620 19:05:48.468337 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:48.468340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:48.468464 (+ 124us) server_negotiation.cc:407] Connection header received
0620 19:05:48.468623 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:48.468627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:48.468687 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:48.468785 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:48.469406 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:48.470352 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:48.471482 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:48.471863 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:48.471988 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:48.476186 (+ 4198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:46179: BlockingRecv error: recv got EOF from 127.1.154.129:46179 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":60,"threads_started":1}
Jun 20 19:05:48 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:48 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:48.677618 10841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:48.670744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:38599)
0620 19:05:48.671024 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:48.671037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:48.671134 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:48.671545 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:48.671549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:48.671788 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:48.671933 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:48.671939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:48.672797 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:48.672800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:48.673818 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:48.673831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:48.674456 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:48.674459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:48.674531 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:48.675240 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:48.675253 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:48.677459 (+ 2206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":112,"threads_started":1}
W20260620 19:05:48.678009 10840 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:48.670875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:38599 (local address 127.1.154.190:39569)
0620 19:05:48.671029 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:48.671032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:48.671113 (+ 81us) server_negotiation.cc:407] Connection header received
0620 19:05:48.671199 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:48.671201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:48.671242 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:48.671320 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:48.672101 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:48.672625 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:48.674007 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:48.674292 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:48.674422 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:48.677844 (+ 3422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:38599: BlockingRecv error: recv got EOF from 127.1.154.130:38599 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 19:05:48 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:48 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:48.892163 10842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:48.886385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:44303)
0620 19:05:48.886773 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:48.886790 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:48.886929 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:48.887302 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:48.887305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:48.887524 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:48.887609 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:48.887614 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:48.888366 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:48.888372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:48.888921 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:48.888930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:48.889367 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:48.889370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:48.889458 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:48.890144 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:48.890162 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:48.892042 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":145,"threads_started":1}
W20260620 19:05:48.892589 10840 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:48.886592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:44303 (local address 127.1.154.190:39569)
0620 19:05:48.886773 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:48.886778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:48.886872 (+ 94us) server_negotiation.cc:407] Connection header received
0620 19:05:48.887041 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:48.887044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:48.887111 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:48.887282 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:48.887726 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:48.888254 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:48.889038 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:48.889255 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:48.889363 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:48.892428 (+ 3065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:44303: BlockingRecv error: recv got EOF from 127.1.154.131:44303 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 19:05:49 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:49 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:49.483526 10843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:49.477136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:49251)
0620 19:05:49.477402 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:49.477415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:49.477501 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:49.477822 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:49.477825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:49.478004 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:49.478099 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:49.478104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:49.479017 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:49.479019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:49.479578 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:49.479585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:49.480336 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:49.480339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:49.480433 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:49.481218 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:49.481230 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:49.483394 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":101,"threads_started":1}
W20260620 19:05:49.484002 10844 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:49.477282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:49251 (local address 127.1.154.190:39569)
0620 19:05:49.477527 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:49.477531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:49.477543 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:05:49.477577 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:49.477579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:49.477641 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:49.477723 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:49.478242 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:49.478856 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:49.479822 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:49.480144 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:49.480309 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:49.483763 (+ 3454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:49251: BlockingRecv error: recv got EOF from 127.1.154.129:49251 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":98,"threads_started":1}
Jun 20 19:05:49 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:49 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:49.685494 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:49.679004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:41759)
0620 19:05:49.679319 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:49.679334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:49.679432 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:49.679776 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:49.679779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:49.679965 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:49.680070 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:49.680075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:49.680935 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:49.680939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:49.681938 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:49.681952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:49.682484 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:49.682487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:49.682556 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:49.683315 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:49.683330 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:49.685362 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":115,"threads_started":1}
W20260620 19:05:49.685669 10844 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:49.679056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:41759 (local address 127.1.154.190:39569)
0620 19:05:49.679360 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:49.679365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:49.679390 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:05:49.679582 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:49.679585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:49.679636 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:49.679722 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:49.680199 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:49.680760 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:49.682128 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:49.682350 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:49.682446 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:49.685516 (+ 3070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:41759: BlockingRecv error: recv got EOF from 127.1.154.130:41759 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
Jun 20 19:05:49 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:49 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:49.898861 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:49.893376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:60683)
0620 19:05:49.893681 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:49.893693 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:49.893838 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:49.894133 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:49.894136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:49.894348 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:49.894440 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:49.894446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:49.895200 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:49.895206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:49.895772 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:49.895779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:49.896238 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:49.896242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:49.896323 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:49.896841 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:49.896852 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:49.898689 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
W20260620 19:05:49.898923 10844 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:49.893377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:60683 (local address 127.1.154.190:39569)
0620 19:05:49.893556 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:49.893561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:49.893782 (+ 221us) server_negotiation.cc:407] Connection header received
0620 19:05:49.893971 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:49.893974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:49.894026 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:49.894114 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:49.894559 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:49.895089 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:49.895903 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:49.896122 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:49.896234 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:49.898807 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:60683: BlockingRecv error: recv got EOF from 127.1.154.131:60683 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 19:05:50 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:50 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:50.491135 10847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:50.484600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:47885)
0620 19:05:50.485010 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:50.485033 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:50.485174 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:50.485689 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:50.485693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:50.485912 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:50.486029 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:50.486035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:50.486875 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:50.486877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:50.487603 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:50.487612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:50.488177 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:50.488180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:50.488240 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:50.488887 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:50.488898 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:50.490952 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":106,"threads_started":1}
W20260620 19:05:50.491294 10848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:50.484814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:47885 (local address 127.1.154.190:39569)
0620 19:05:50.485151 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:50.485156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:50.485177 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:05:50.485343 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:50.485346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:50.485406 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:50.485510 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:50.486165 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:50.486754 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:50.487775 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:50.488019 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:50.488151 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:50.491107 (+ 2956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:47885: BlockingRecv error: recv got EOF from 127.1.154.129:47885 (error 108)
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":120,"threads_started":1}
Jun 20 19:05:50 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:50 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:50.693449 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:50.686459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:45119)
0620 19:05:50.686876 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:50.686897 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:50.687082 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:50.687749 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:50.687752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:50.687999 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:50.688124 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:50.688129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:50.688992 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:50.688995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:50.689818 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:50.689827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:50.690374 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:50.690376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:50.690475 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:50.691117 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:50.691131 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:50.693253 (+ 2122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":82,"threads_started":1}
W20260620 19:05:50.693470 10848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:50.686755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:45119 (local address 127.1.154.190:39569)
0620 19:05:50.686929 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:50.686937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:50.686996 (+ 59us) server_negotiation.cc:407] Connection header received
0620 19:05:50.687393 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:50.687396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:50.687480 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:50.687570 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:50.688308 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:50.688851 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:50.689993 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:50.690214 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:50.690313 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:50.693371 (+ 3058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:45119: BlockingRecv error: recv got EOF from 127.1.154.130:45119 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 19:05:50 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:50 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:50.906787 10850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:50.900182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:33307)
0620 19:05:50.900565 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:50.900581 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:50.900716 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:50.901063 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:50.901066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:50.901269 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:50.901376 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:50.901382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:50.902320 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:50.902323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:50.903263 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:50.903275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:50.903946 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:50.903951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:50.904038 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:50.904653 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:50.904673 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:50.906608 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":150,"threads_started":1}
W20260620 19:05:50.906848 10848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:50.900306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:33307 (local address 127.1.154.190:39569)
0620 19:05:50.900530 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:50.900538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:50.900714 (+ 176us) server_negotiation.cc:407] Connection header received
0620 19:05:50.900815 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:50.900818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:50.900881 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:50.900978 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:50.901540 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:50.902163 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:50.903456 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:50.903794 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:50.903899 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:50.906734 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:33307: BlockingRecv error: recv got EOF from 127.1.154.131:33307 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 19:05:51 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:51 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:51.502370 10851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:51.492186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:52401)
0620 19:05:51.492466 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:51.492480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:51.492569 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:51.493076 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:51.493079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:51.493286 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:51.493375 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:51.493380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:51.494195 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:51.494202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:51.494757 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:51.494765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:51.495240 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:51.495244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:51.495324 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:51.499595 (+ 4271us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:51.499617 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:51.502237 (+ 2620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":110,"threads_started":1}
W20260620 19:05:51.502738 10852 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:51.492515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:52401 (local address 127.1.154.190:39569)
0620 19:05:51.492862 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:51.492865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:51.492877 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:05:51.492910 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:51.492912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:51.492953 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:51.493059 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:51.493495 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:51.494085 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:51.494884 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:51.495109 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:51.495227 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:51.502557 (+ 7330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:52401: BlockingRecv error: recv got EOF from 127.1.154.129:52401 (error 108)
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":88,"threads_started":1}
Jun 20 19:05:51 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:51 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:51.701404 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:51.694418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:33585)
0620 19:05:51.694703 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:51.694715 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:51.694834 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:51.695112 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:51.695115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:51.695454 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:51.695633 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:51.695643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:51.696564 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:51.696567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:51.697259 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:51.697267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:51.697804 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:51.697807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:51.697877 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:51.698583 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:51.698599 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:51.701261 (+ 2662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":117,"threads_started":1}
W20260620 19:05:51.701515 10852 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:51.694568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:33585 (local address 127.1.154.190:39569)
0620 19:05:51.694706 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:51.694710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:51.694812 (+ 102us) server_negotiation.cc:407] Connection header received
0620 19:05:51.694933 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:51.694935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:51.694990 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:51.695098 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:51.695811 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:51.696390 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:51.697389 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:51.697671 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:51.697761 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:51.701420 (+ 3659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:33585: BlockingRecv error: recv got EOF from 127.1.154.130:33585 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 19:05:51 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:51 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.131@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:51.913321 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:51.907697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.131:36721)
0620 19:05:51.908137 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:51.908153 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:51.908292 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:51.908569 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:51.908572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:51.908790 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:51.908890 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:51.908895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:51.909702 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:51.909706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:51.910279 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:51.910286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:51.910746 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:51.910749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:51.910829 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:51.911379 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:51.911390 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:51.913137 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":114,"threads_started":1}
W20260620 19:05:51.913373 10852 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:51.907927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:36721 (local address 127.1.154.190:39569)
0620 19:05:51.908117 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:51.908122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:51.908238 (+ 116us) server_negotiation.cc:407] Connection header received
0620 19:05:51.908406 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:51.908410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:51.908464 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:51.908550 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:51.909011 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:51.909529 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:51.910403 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:51.910633 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:51.910741 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:51.913254 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.131:36721: BlockingRecv error: recv got EOF from 127.1.154.131:36721 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 20 19:05:52 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:52 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.129@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:52.508905 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:52.503326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.129:54515)
0620 19:05:52.503640 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:52.503653 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:52.503766 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:52.504019 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:52.504021 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:52.504202 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:52.504292 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:52.504298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:52.505040 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:52.505043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:52.505581 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:52.505588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:52.506089 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:52.506092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:52.506150 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:52.506641 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:52.506652 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:52.508758 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":94,"threads_started":1}
W20260620 19:05:52.509209 10856 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:52.503467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:54515 (local address 127.1.154.190:39569)
0620 19:05:52.503681 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:52.503685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:52.503715 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:05:52.503867 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:52.503869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:52.503911 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:52.503988 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:52.504401 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:52.504938 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:52.505738 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:52.505960 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:52.506048 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:52.508955 (+ 2907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.129:54515: BlockingRecv error: recv got EOF from 127.1.154.129:54515 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":54,"threads_started":1}
Jun 20 19:05:52 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:05:52 dist-test-slave-6wrf krb5kdc[9797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.154.130@KRBTEST.COM for oryx/127.1.154.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:05:52.708478 10857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:52.702439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39569 (local address 127.1.154.130:45475)
0620 19:05:52.702872 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:52.703044 (+ 172us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:52.703186 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:52.703514 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:52.703517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:52.703742 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:52.703847 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:52.703852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:52.704644 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:52.704650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:52.705230 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:52.705238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:52.705745 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:52.705748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:52.705839 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:52.706396 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:52.706408 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:52.708292 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.154.190:39569: Server oryx/127.1.154.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":71,"threads_started":1}
W20260620 19:05:52.708567 10856 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:52.702711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:45475 (local address 127.1.154.190:39569)
0620 19:05:52.702876 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:52.702883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:52.703135 (+ 252us) server_negotiation.cc:407] Connection header received
0620 19:05:52.703324 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:52.703327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:52.703396 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:52.703495 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:52.703979 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:52.704534 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:52.705365 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:52.705640 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:52.705729 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:52.708447 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.154.130:45475: BlockingRecv error: recv got EOF from 127.1.154.130:45475 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 19:05:52.794749 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 10380
I20260620 19:05:52.800678 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 10533
I20260620 19:05:52.806797 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 10665
I20260620 19:05:52.812001 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 10310
2026-06-20T19:05:52Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18331 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T19:05:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:52Z Disabled control of system clock
I20260620 19:05:52.842473 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:39327
--webserver_interface=127.1.154.190
--webserver_port=0
--builtin_ntp_servers=127.1.154.148:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:39327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 19:05:52.956573 10861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:52.956897 10861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:52.956935 10861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:52.961062 10861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:52.961136 10861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:52.961153 10861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:52.961169 10861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:52.961185 10861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:52.965880 10861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:39327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:39327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:52.966929 10861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:52.968325 10861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:52.978591 10867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:52.978582 10866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:52.978637 10869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:52.978981 10861 server_base.cc:1034] running on GCE node
I20260620 19:05:52.979537 10861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:52.980698 10861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:52.982079 10861 hybrid_clock.cc:648] HybridClock initialized: now 1781982352982064 us; error 60 us; skew 500 ppm
I20260620 19:05:52.984429 10861 webserver.cc:460] Webserver started at http://127.1.154.190:41267/ using document root <none> and password file <none>
I20260620 19:05:52.985147 10861 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:52.985246 10861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:52.985479 10861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:52.987624 10861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "d52017c4906a4941b4e11aa226a48005"
format_stamp: "Formatted at 2026-06-20 19:05:52 on dist-test-slave-6wrf"
server_key: "b5ab68040d143746dbad9d3cc584a6e9"
server_key_iv: "5efdcde9e27f0011e80c9c6fee417884"
server_key_version: "encryptionkey@0"
I20260620 19:05:52.988191 10861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "d52017c4906a4941b4e11aa226a48005"
format_stamp: "Formatted at 2026-06-20 19:05:52 on dist-test-slave-6wrf"
server_key: "b5ab68040d143746dbad9d3cc584a6e9"
server_key_iv: "5efdcde9e27f0011e80c9c6fee417884"
server_key_version: "encryptionkey@0"
I20260620 19:05:52.992022 10861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:05:52.994544 10875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:52.995973 10861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:52.996150 10861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "d52017c4906a4941b4e11aa226a48005"
format_stamp: "Formatted at 2026-06-20 19:05:52 on dist-test-slave-6wrf"
server_key: "b5ab68040d143746dbad9d3cc584a6e9"
server_key_iv: "5efdcde9e27f0011e80c9c6fee417884"
server_key_version: "encryptionkey@0"
I20260620 19:05:52.996271 10861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:53.009831 10861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:53.010562 10861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:53.010762 10861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:53.017795 10861 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:39327
I20260620 19:05:53.017844 10927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:39327 every 8 connection(s)
I20260620 19:05:53.018744 10861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:53.021667 10928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:53.027024 10928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005: Bootstrap starting.
I20260620 19:05:53.028800 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 10861
I20260620 19:05:53.028960 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:53.029232 1642 external_mini_cluster.cc:1428] Setting key 9f81422e273e1d6cf187b716efae8cc3
I20260620 19:05:53.029541 10928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:53.030660 10928 log.cc:824] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:53.032639 10928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005: No bootstrap required, opened a new log
I20260620 19:05:53.036715 10928 raft_consensus.cc:348] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52017c4906a4941b4e11aa226a48005" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39327 } }
I20260620 19:05:53.037030 10928 raft_consensus.cc:374] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:53.037092 10928 raft_consensus.cc:729] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d52017c4906a4941b4e11aa226a48005, State: Initialized, Role: FOLLOWER
I20260620 19:05:53.038110 10928 consensus_queue.cc:260] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52017c4906a4941b4e11aa226a48005" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39327 } }
I20260620 19:05:53.038443 10928 raft_consensus.cc:388] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:53.038594 10928 raft_consensus.cc:482] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:53.038753 10928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:53.040150 10928 raft_consensus.cc:504] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52017c4906a4941b4e11aa226a48005" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39327 } }
I20260620 19:05:53.040263 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:53.030569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.1.154.190:39327)
0620 19:05:53.031262 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:53.031278 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:53.031311 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:05:53.031960 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:53.031976 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:53.032280 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:53.032540 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:53.032947 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:53.033984 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:53.035310 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:53.036058 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:53.036270 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:53.037465 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:53.037473 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:53.037492 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:53.038105 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:53.038191 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:53.038218 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:53.038324 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:53.038613 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 19:05:53.039158 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":192,"threads_started":1}
I20260620 19:05:53.040781 10928 leader_election.cc:302] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d52017c4906a4941b4e11aa226a48005; no voters:
I20260620 19:05:53.041396 10928 leader_election.cc:288] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:53.041795 10933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:53.042119 10933 raft_consensus.cc:686] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [term 1 LEADER]: Becoming Leader. State: Replica: d52017c4906a4941b4e11aa226a48005, State: Running, Role: LEADER
I20260620 19:05:53.042501 10933 consensus_queue.cc:237] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52017c4906a4941b4e11aa226a48005" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39327 } }
I20260620 19:05:53.043265 10928 sys_catalog.cc:564] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:53.044332 10935 sys_catalog.cc:455] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d52017c4906a4941b4e11aa226a48005. Latest consensus state: current_term: 1 leader_uuid: "d52017c4906a4941b4e11aa226a48005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52017c4906a4941b4e11aa226a48005" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39327 } } }
I20260620 19:05:53.044308 10934 sys_catalog.cc:455] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d52017c4906a4941b4e11aa226a48005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52017c4906a4941b4e11aa226a48005" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39327 } } }
I20260620 19:05:53.044503 10934 sys_catalog.cc:458] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:53.044503 10935 sys_catalog.cc:458] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:53.045006 10941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:53.049059 10941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:53.056520 10941 catalog_manager.cc:1347] Generated new cluster ID: 2a98395ddf804e7d8d9f6307d6adcd92
I20260620 19:05:53.056634 10941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:53.065577 10941 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:53.066653 10941 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:53.074196 10941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d52017c4906a4941b4e11aa226a48005: Generated new TSK 0
I20260620 19:05:53.075381 10941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:05:53.088621 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39327
--builtin_ntp_servers=127.1.154.148:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 19:05:53.206391 10952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:53.206656 10952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:53.206694 10952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:53.210870 10952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:53.210925 10952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:53.210983 10952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:05:53.215795 10952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:53.216897 10952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:53.218281 10952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:53.228631 10960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:53.228603 10957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:53.228610 10958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:53.229074 10952 server_base.cc:1034] running on GCE node
I20260620 19:05:53.229477 10952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:53.230152 10952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:53.231348 10952 hybrid_clock.cc:648] HybridClock initialized: now 1781982353231330 us; error 33 us; skew 500 ppm
I20260620 19:05:53.233237 10952 webserver.cc:460] Webserver started at http://127.1.154.129:32837/ using document root <none> and password file <none>
I20260620 19:05:53.233911 10952 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:53.234005 10952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:53.234228 10952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:53.236346 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "12c9f1eabd9f4f8e9a3250e1473fbafb"
format_stamp: "Formatted at 2026-06-20 19:05:53 on dist-test-slave-6wrf"
server_key: "512a6b296d19a7fbdd7b33f22ddc03d9"
server_key_iv: "c90aa88bc5b76628e018e1e938e130f3"
server_key_version: "encryptionkey@0"
I20260620 19:05:53.236931 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "12c9f1eabd9f4f8e9a3250e1473fbafb"
format_stamp: "Formatted at 2026-06-20 19:05:53 on dist-test-slave-6wrf"
server_key: "512a6b296d19a7fbdd7b33f22ddc03d9"
server_key_iv: "c90aa88bc5b76628e018e1e938e130f3"
server_key_version: "encryptionkey@0"
I20260620 19:05:53.240769 10952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:05:53.243860 10966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:53.245309 10952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:53.245456 10952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "12c9f1eabd9f4f8e9a3250e1473fbafb"
format_stamp: "Formatted at 2026-06-20 19:05:53 on dist-test-slave-6wrf"
server_key: "512a6b296d19a7fbdd7b33f22ddc03d9"
server_key_iv: "c90aa88bc5b76628e018e1e938e130f3"
server_key_version: "encryptionkey@0"
I20260620 19:05:53.245586 10952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:53.262470 10952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:53.263228 10952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:53.263443 10952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:53.264046 10952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:53.265030 10952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:53.265121 10952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:53.265187 10952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:53.265216 10952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:53.274698 10952 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:39387
I20260620 19:05:53.274724 11079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:39387 every 8 connection(s)
I20260620 19:05:53.275691 10952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:05:53.285346 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:53.277813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:46995 (local address 127.1.154.190:39327)
0620 19:05:53.278144 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:53.278148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:53.279249 (+ 1101us) server_negotiation.cc:407] Connection header received
0620 19:05:53.280632 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:53.280635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:53.280692 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:53.280796 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:53.281709 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:53.282287 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:53.283420 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:53.283653 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:53.283774 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:53.284856 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:53.284866 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:53.284868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:53.284879 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:53.284898 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:53.284905 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:53.284980 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:53.285141 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 19:05:53.285197 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260620 19:05:53.285813 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 10952
I20260620 19:05:53.285933 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:05:53.286243 1642 external_mini_cluster.cc:1428] Setting key 7b00410347338dd1f75119d807f629f3
I20260620 19:05:53.286451 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:53.278465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39327 (local address 127.1.154.129:46995)
0620 19:05:53.279097 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:53.279137 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:53.280423 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:53.280966 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:53.280973 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:53.281050 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:53.281512 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:53.281524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:53.282417 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:53.282420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:53.283293 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:53.283302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:53.283807 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:53.283810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:53.283994 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:53.284603 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:05:53.284605 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:53.284643 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:05:53.284703 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:53.284992 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:53.285004 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:53.285011 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 19:05:53.285140 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 19:05:53.285438 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":119,"threads_started":1}
I20260620 19:05:53.287685 11080 heartbeater.cc:347] Connected to a master server at 127.1.154.190:39327
I20260620 19:05:53.287921 11080 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:53.288137 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39327
--builtin_ntp_servers=127.1.154.148:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 19:05:53.288470 11080 heartbeater.cc:511] Master 127.1.154.190:39327 requested a full tablet report, sending...
I20260620 19:05:53.291177 10889 ts_manager.cc:194] Registered new tserver with Master: 12c9f1eabd9f4f8e9a3250e1473fbafb (127.1.154.129:39387)
I20260620 19:05:53.292953 10889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.154.129:46995
W20260620 19:05:53.410256 11083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:53.410540 11083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:53.410579 11083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:53.414608 11083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:53.414669 11083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:53.414724 11083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:53.419675 11083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:53.420832 11083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:53.422219 11083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:53.431752 11091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:53.431856 11089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:53.431856 11088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:53.432251 11083 server_base.cc:1034] running on GCE node
I20260620 19:05:53.432698 11083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:53.433348 11083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:53.434548 11083 hybrid_clock.cc:648] HybridClock initialized: now 1781982353434537 us; error 78 us; skew 500 ppm
I20260620 19:05:53.436434 11083 webserver.cc:460] Webserver started at http://127.1.154.130:34751/ using document root <none> and password file <none>
I20260620 19:05:53.437424 11083 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:53.437675 11083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:53.437947 11083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:53.440027 11083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "66f44f378181418c9823c479ab205af3"
format_stamp: "Formatted at 2026-06-20 19:05:53 on dist-test-slave-6wrf"
server_key: "a3a22c91448246b04f5790d72c252d34"
server_key_iv: "f8bdb0b621eeac80eaad064b55dc6cc3"
server_key_version: "encryptionkey@0"
I20260620 19:05:53.440620 11083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "66f44f378181418c9823c479ab205af3"
format_stamp: "Formatted at 2026-06-20 19:05:53 on dist-test-slave-6wrf"
server_key: "a3a22c91448246b04f5790d72c252d34"
server_key_iv: "f8bdb0b621eeac80eaad064b55dc6cc3"
server_key_version: "encryptionkey@0"
I20260620 19:05:53.444945 11083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 19:05:53.447916 11097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:53.449532 11083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:53.449724 11083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "66f44f378181418c9823c479ab205af3"
format_stamp: "Formatted at 2026-06-20 19:05:53 on dist-test-slave-6wrf"
server_key: "a3a22c91448246b04f5790d72c252d34"
server_key_iv: "f8bdb0b621eeac80eaad064b55dc6cc3"
server_key_version: "encryptionkey@0"
I20260620 19:05:53.449884 11083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:53.474277 11083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:53.475513 11083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:53.475970 11083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:53.476816 11083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:53.477907 11083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:53.477957 11083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:53.478001 11083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:53.478017 11083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:53.488843 11083 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:43657
I20260620 19:05:53.488916 11210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:43657 every 8 connection(s)
I20260620 19:05:53.490584 11083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:53.495570 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 11083
I20260620 19:05:53.495705 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:05:53.496022 1642 external_mini_cluster.cc:1428] Setting key 898806bb6ea86c9a657dbafd060f071e
I20260620 19:05:53.498250 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39327
--builtin_ntp_servers=127.1.154.148:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 19:05:53.502837 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:53.492901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:53055 (local address 127.1.154.190:39327)
0620 19:05:53.493255 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:53.493260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:53.494321 (+ 1061us) server_negotiation.cc:407] Connection header received
0620 19:05:53.495940 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:53.495943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:53.495995 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:53.496076 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:53.497164 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:53.497826 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:53.499638 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:53.499923 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:53.500016 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:53.502050 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:53.502054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:53.502057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:53.502074 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:53.502099 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:53.502109 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:53.502200 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:53.502552 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 19:05:53.502650 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260620 19:05:53.504295 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:53.493343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39327 (local address 127.1.154.130:53055)
0620 19:05:53.494162 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:53.494205 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:53.495630 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:53.496289 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:53.496298 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:53.496402 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:53.497008 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:53.497027 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:53.497957 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:53.497962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:53.499470 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:53.499485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:53.500193 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:53.500196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:53.500339 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:53.501494 (+ 1155us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:05:53.501496 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:53.501577 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:05:53.501775 (+ 198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:53.502218 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:53.502230 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:53.502241 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 19:05:53.502549 (+ 308us) client_negotiation.cc:240] Negotiation successful
0620 19:05:53.503033 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":131,"threads_started":1}
I20260620 19:05:53.506213 11211 heartbeater.cc:347] Connected to a master server at 127.1.154.190:39327
I20260620 19:05:53.506559 11211 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:53.507392 11211 heartbeater.cc:511] Master 127.1.154.190:39327 requested a full tablet report, sending...
I20260620 19:05:53.508925 10889 ts_manager.cc:194] Registered new tserver with Master: 66f44f378181418c9823c479ab205af3 (127.1.154.130:43657)
I20260620 19:05:53.509871 10889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.154.130:53055
W20260620 19:05:53.617056 11214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:53.617334 11214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:53.617372 11214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:53.621472 11214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:53.621538 11214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:53.621649 11214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:53.626545 11214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:39327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:53.627734 11214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:53.629066 11214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:53.639258 11219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:53.639262 11220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:53.639257 11222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:53.639549 11214 server_base.cc:1034] running on GCE node
I20260620 19:05:53.640108 11214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:53.640712 11214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:53.642057 11214 hybrid_clock.cc:648] HybridClock initialized: now 1781982353642041 us; error 50 us; skew 500 ppm
I20260620 19:05:53.644246 11214 webserver.cc:460] Webserver started at http://127.1.154.131:43251/ using document root <none> and password file <none>
I20260620 19:05:53.645027 11214 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:53.645108 11214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:53.645334 11214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:53.647728 11214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "1e7b4462e50a437c8dc26781c0b1e536"
format_stamp: "Formatted at 2026-06-20 19:05:53 on dist-test-slave-6wrf"
server_key: "7f3b67d9b8c917ade972d6f4f6a582d5"
server_key_iv: "094aa1a9cf7af94239ac7fad04414da9"
server_key_version: "encryptionkey@0"
I20260620 19:05:53.648355 11214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "1e7b4462e50a437c8dc26781c0b1e536"
format_stamp: "Formatted at 2026-06-20 19:05:53 on dist-test-slave-6wrf"
server_key: "7f3b67d9b8c917ade972d6f4f6a582d5"
server_key_iv: "094aa1a9cf7af94239ac7fad04414da9"
server_key_version: "encryptionkey@0"
I20260620 19:05:53.652411 11214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:05:53.655445 11228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:53.657114 11214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:05:53.657225 11214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "1e7b4462e50a437c8dc26781c0b1e536"
format_stamp: "Formatted at 2026-06-20 19:05:53 on dist-test-slave-6wrf"
server_key: "7f3b67d9b8c917ade972d6f4f6a582d5"
server_key_iv: "094aa1a9cf7af94239ac7fad04414da9"
server_key_version: "encryptionkey@0"
I20260620 19:05:53.657410 11214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:53.683907 11214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:53.684680 11214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:53.684906 11214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:53.685551 11214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:53.686583 11214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:53.686659 11214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:53.686741 11214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:53.686789 11214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:53.699287 11214 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:39615
I20260620 19:05:53.699716 11341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:39615 every 8 connection(s)
I20260620 19:05:53.700608 11214 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:53.708084 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 11214
I20260620 19:05:53.708276 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:05:53.708635 1642 external_mini_cluster.cc:1428] Setting key 55114df392e33d87c358fcdedc8fa8ff
I20260620 19:05:53.710929 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:53.702580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:53645 (local address 127.1.154.190:39327)
0620 19:05:53.702729 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:53.702735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:53.703913 (+ 1178us) server_negotiation.cc:407] Connection header received
0620 19:05:53.705388 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:53.705393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:53.705466 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:53.705582 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:53.706568 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:53.707198 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:53.708487 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:53.708864 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:53.708988 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:53.710385 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:53.710390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:53.710393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:53.710407 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:53.710432 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:53.710445 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:53.710556 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:53.710689 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 19:05:53.710755 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:05:53.711864 11344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:53.703019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39327 (local address 127.1.154.131:53645)
0620 19:05:53.703778 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:53.703817 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:53.705089 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:53.705803 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:53.705809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:53.705889 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:53.706393 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:53.706405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:53.707382 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:53.707385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:53.708316 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:53.708333 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:53.709116 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:53.709121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:53.709319 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:53.710049 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:05:53.710051 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:53.710112 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:05:53.710218 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:53.710543 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:53.710550 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:53.710555 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:05:53.710656 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 19:05:53.710968 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":120,"threads_started":1}
I20260620 19:05:53.713235 11342 heartbeater.cc:347] Connected to a master server at 127.1.154.190:39327
I20260620 19:05:53.713443 11342 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:53.713999 11342 heartbeater.cc:511] Master 127.1.154.190:39327 requested a full tablet report, sending...
I20260620 19:05:53.715312 10889 ts_manager.cc:194] Registered new tserver with Master: 1e7b4462e50a437c8dc26781c0b1e536 (127.1.154.131:39615)
I20260620 19:05:53.716161 10889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.154.131:53645
I20260620 19:05:53.723776 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:05:53.726881 10931 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:53.726175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.1.154.190:39327)
0620 19:05:53.726328 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:53.726332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:53.726345 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:05:53.726382 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:53.726384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:53.726425 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:53.726499 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:53.726767 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56886: BlockingRecv error: recv got EOF from 127.0.0.1:56886 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 19:05:53.727056 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 10952
I20260620 19:05:53.733407 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 11083
I20260620 19:05:53.739338 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 11214
I20260620 19:05:53.745467 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 10861
2026-06-20T19:05:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (934 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T19:05:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:53Z Disabled control of system clock
I20260620 19:05:53.777751 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43783
--webserver_interface=127.1.154.190
--webserver_port=0
--builtin_ntp_servers=127.1.154.148:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:05:53.888856 11353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:53.889137 11353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:53.889173 11353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:53.893357 11353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:53.893427 11353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:53.893443 11353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:53.893461 11353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:53.893476 11353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:53.898356 11353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:43783
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:53.899507 11353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:53.900882 11353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:53.909996 11361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:53.910148 11358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:53.910284 11353 server_base.cc:1034] running on GCE node
W20260620 19:05:53.910153 11359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:53.911296 11353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:53.912441 11353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:53.913703 11353 hybrid_clock.cc:648] HybridClock initialized: now 1781982353913669 us; error 52 us; skew 500 ppm
I20260620 19:05:53.915795 11353 webserver.cc:460] Webserver started at http://127.1.154.190:44313/ using document root <none> and password file <none>
I20260620 19:05:53.916399 11353 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:53.916469 11353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:53.916692 11353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:53.918895 11353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "ca5a759a87b74f62921f0dc1f05aaaaf"
format_stamp: "Formatted at 2026-06-20 19:05:53 on dist-test-slave-6wrf"
server_key: "36776067e3138d831e54919cedbd5817"
server_key_iv: "6f8c88ecbf4a5b80ac1f6c757b852359"
server_key_version: "encryptionkey@0"
I20260620 19:05:53.919480 11353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "ca5a759a87b74f62921f0dc1f05aaaaf"
format_stamp: "Formatted at 2026-06-20 19:05:53 on dist-test-slave-6wrf"
server_key: "36776067e3138d831e54919cedbd5817"
server_key_iv: "6f8c88ecbf4a5b80ac1f6c757b852359"
server_key_version: "encryptionkey@0"
I20260620 19:05:53.923429 11353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:05:53.926463 11367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:53.928447 11353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:53.928692 11353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "ca5a759a87b74f62921f0dc1f05aaaaf"
format_stamp: "Formatted at 2026-06-20 19:05:53 on dist-test-slave-6wrf"
server_key: "36776067e3138d831e54919cedbd5817"
server_key_iv: "6f8c88ecbf4a5b80ac1f6c757b852359"
server_key_version: "encryptionkey@0"
I20260620 19:05:53.928860 11353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:53.934993 11353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:53.935734 11353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:53.935961 11353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:53.944239 11353 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:43783
I20260620 19:05:53.944262 11419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:43783 every 8 connection(s)
I20260620 19:05:53.945537 11353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:53.949062 11420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:53.953971 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 11353
I20260620 19:05:53.954116 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:53.954375 1642 external_mini_cluster.cc:1428] Setting key 1c5d4a4dc939a7a9347ebbb6c797723d
I20260620 19:05:53.955006 11420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf: Bootstrap starting.
I20260620 19:05:53.957489 11420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:53.958393 11420 log.cc:824] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:53.960266 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:53.955795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.1.154.190:43783)
0620 19:05:53.956437 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:53.956452 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:53.956487 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:05:53.957171 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:53.957188 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:53.957523 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:53.957933 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:53.958736 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:53.958741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:53.958751 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:53.958761 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:53.958849 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:53.958874 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:53.958992 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:53.959148 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 19:05:53.959399 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":134,"threads_started":1}
I20260620 19:05:53.960647 11420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf: No bootstrap required, opened a new log
I20260620 19:05:53.964322 11420 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5a759a87b74f62921f0dc1f05aaaaf" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43783 } }
I20260620 19:05:53.964548 11420 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:53.964581 11420 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5a759a87b74f62921f0dc1f05aaaaf, State: Initialized, Role: FOLLOWER
I20260620 19:05:53.964978 11420 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5a759a87b74f62921f0dc1f05aaaaf" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43783 } }
I20260620 19:05:53.965127 11420 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:53.965209 11420 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:53.965341 11420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:53.966329 11420 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5a759a87b74f62921f0dc1f05aaaaf" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43783 } }
I20260620 19:05:53.966737 11420 leader_election.cc:302] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca5a759a87b74f62921f0dc1f05aaaaf; no voters:
I20260620 19:05:53.967054 11420 leader_election.cc:288] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:53.967263 11425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:53.967674 11425 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [term 1 LEADER]: Becoming Leader. State: Replica: ca5a759a87b74f62921f0dc1f05aaaaf, State: Running, Role: LEADER
I20260620 19:05:53.968003 11425 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5a759a87b74f62921f0dc1f05aaaaf" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43783 } }
I20260620 19:05:53.968325 11420 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:53.969583 11426 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca5a759a87b74f62921f0dc1f05aaaaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5a759a87b74f62921f0dc1f05aaaaf" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43783 } } }
I20260620 19:05:53.969725 11426 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:53.969758 11427 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca5a759a87b74f62921f0dc1f05aaaaf. Latest consensus state: current_term: 1 leader_uuid: "ca5a759a87b74f62921f0dc1f05aaaaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5a759a87b74f62921f0dc1f05aaaaf" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43783 } } }
I20260620 19:05:53.969861 11427 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:53.970144 11434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:53.972734 11434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:53.978793 11434 catalog_manager.cc:1347] Generated new cluster ID: 500b9a4cb1f146e2af095a47dc108d6b
I20260620 19:05:53.978874 11434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:54.002547 11434 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:54.002889 11434 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:54.010980 11434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca5a759a87b74f62921f0dc1f05aaaaf: Generated new TSK 0
I20260620 19:05:54.011818 11434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:05:54.018697 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43783
--builtin_ntp_servers=127.1.154.148:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:05:54.135540 11444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:54.135860 11444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:54.135936 11444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:54.140010 11444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:54.140105 11444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:54.140197 11444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:05:54.145537 11444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:54.146718 11444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:54.148119 11444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:54.158397 11452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:54.158465 11450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:54.158396 11449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:54.158988 11444 server_base.cc:1034] running on GCE node
I20260620 19:05:54.159389 11444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:54.160032 11444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:54.161229 11444 hybrid_clock.cc:648] HybridClock initialized: now 1781982354161212 us; error 36 us; skew 500 ppm
I20260620 19:05:54.163862 11444 webserver.cc:460] Webserver started at http://127.1.154.129:41377/ using document root <none> and password file <none>
I20260620 19:05:54.164734 11444 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:54.164805 11444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:54.165030 11444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:54.167506 11444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "315d3175afe84b35986e1ccf595a1186"
format_stamp: "Formatted at 2026-06-20 19:05:54 on dist-test-slave-6wrf"
server_key: "b67853c52af834be718b9147ca91436c"
server_key_iv: "bece1a736c7e2c09a40806235845e051"
server_key_version: "encryptionkey@0"
I20260620 19:05:54.168443 11444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "315d3175afe84b35986e1ccf595a1186"
format_stamp: "Formatted at 2026-06-20 19:05:54 on dist-test-slave-6wrf"
server_key: "b67853c52af834be718b9147ca91436c"
server_key_iv: "bece1a736c7e2c09a40806235845e051"
server_key_version: "encryptionkey@0"
I20260620 19:05:54.173843 11444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 19:05:54.177160 11458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:54.178978 11444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:54.179141 11444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "315d3175afe84b35986e1ccf595a1186"
format_stamp: "Formatted at 2026-06-20 19:05:54 on dist-test-slave-6wrf"
server_key: "b67853c52af834be718b9147ca91436c"
server_key_iv: "bece1a736c7e2c09a40806235845e051"
server_key_version: "encryptionkey@0"
I20260620 19:05:54.179279 11444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:54.184234 11444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:54.184887 11444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:54.185078 11444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:54.185702 11444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:54.186679 11444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:54.186726 11444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:54.186764 11444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:54.186812 11444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:54.195796 11444 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:42717
I20260620 19:05:54.195899 11571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:42717 every 8 connection(s)
I20260620 19:05:54.196787 11444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:05:54.203878 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:54.198965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:34955 (local address 127.1.154.190:43783)
0620 19:05:54.199104 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:54.199107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:54.200642 (+ 1535us) server_negotiation.cc:407] Connection header received
0620 19:05:54.202093 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:54.202096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:54.202143 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:54.202255 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:54.203384 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:54.203388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:54.203390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:54.203402 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:54.203422 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:54.203430 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:54.203549 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:54.203700 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 19:05:54.203745 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:05:54.204835 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:54.199645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43783 (local address 127.1.154.129:34955)
0620 19:05:54.200473 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:54.200516 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:54.201881 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:54.202414 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:54.202421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:54.202503 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:54.203133 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:05:54.203135 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:54.203173 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:05:54.203240 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:54.203542 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:54.203549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:54.203555 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:05:54.203741 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 19:05:54.203973 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":137,"threads_started":1}
I20260620 19:05:54.206053 11572 heartbeater.cc:347] Connected to a master server at 127.1.154.190:43783
I20260620 19:05:54.206260 11572 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:54.206604 11572 heartbeater.cc:511] Master 127.1.154.190:43783 requested a full tablet report, sending...
I20260620 19:05:54.207043 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 11444
I20260620 19:05:54.207175 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:05:54.207444 1642 external_mini_cluster.cc:1428] Setting key 9c5279ef00d21e945ba1bb6de0bb6946
I20260620 19:05:54.208824 11384 ts_manager.cc:194] Registered new tserver with Master: 315d3175afe84b35986e1ccf595a1186 (127.1.154.129:42717)
I20260620 19:05:54.210180 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43783
--builtin_ntp_servers=127.1.154.148:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:05:54.327319 11575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:54.327618 11575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:54.327687 11575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:54.333730 11575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:54.333811 11575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:54.333930 11575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:54.340016 11575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:54.341147 11575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:54.342579 11575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:54.353329 11583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:54.353320 11581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:54.353482 11580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:54.353910 11575 server_base.cc:1034] running on GCE node
I20260620 19:05:54.354521 11575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:54.355386 11575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:54.356627 11575 hybrid_clock.cc:648] HybridClock initialized: now 1781982354356586 us; error 53 us; skew 500 ppm
I20260620 19:05:54.359406 11575 webserver.cc:460] Webserver started at http://127.1.154.130:40427/ using document root <none> and password file <none>
I20260620 19:05:54.360354 11575 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:54.360473 11575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:54.360751 11575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:54.363615 11575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "9734afbf344848a9a16bbbd914b00a0a"
format_stamp: "Formatted at 2026-06-20 19:05:54 on dist-test-slave-6wrf"
server_key: "1f015e3b65982696ea6a46a049c3ffaa"
server_key_iv: "84fa96d649d320b62c454d243cb68d22"
server_key_version: "encryptionkey@0"
I20260620 19:05:54.364346 11575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "9734afbf344848a9a16bbbd914b00a0a"
format_stamp: "Formatted at 2026-06-20 19:05:54 on dist-test-slave-6wrf"
server_key: "1f015e3b65982696ea6a46a049c3ffaa"
server_key_iv: "84fa96d649d320b62c454d243cb68d22"
server_key_version: "encryptionkey@0"
I20260620 19:05:54.369856 11575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 19:05:54.373229 11589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:54.375123 11575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:05:54.375303 11575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "9734afbf344848a9a16bbbd914b00a0a"
format_stamp: "Formatted at 2026-06-20 19:05:54 on dist-test-slave-6wrf"
server_key: "1f015e3b65982696ea6a46a049c3ffaa"
server_key_iv: "84fa96d649d320b62c454d243cb68d22"
server_key_version: "encryptionkey@0"
I20260620 19:05:54.375444 11575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:54.380373 11575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:54.381295 11575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:54.381506 11575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:54.382862 11575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:54.384158 11575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:54.384243 11575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:54.384320 11575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:54.384357 11575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:54.395925 11575 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:39469
I20260620 19:05:54.395982 11702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:39469 every 8 connection(s)
I20260620 19:05:54.396908 11575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:54.397315 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 11575
I20260620 19:05:54.397465 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:05:54.397852 1642 external_mini_cluster.cc:1428] Setting key 352b74114fb20cbcc0406c8a63e9d580
I20260620 19:05:54.400476 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43783
--builtin_ntp_servers=127.1.154.148:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 19:05:54.404055 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:54.399105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:49667 (local address 127.1.154.190:43783)
0620 19:05:54.399251 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:54.399255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:54.400590 (+ 1335us) server_negotiation.cc:407] Connection header received
0620 19:05:54.402053 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:54.402056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:54.402096 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:54.402171 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:54.403312 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:54.403318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:54.403324 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:54.403339 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:54.403365 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:54.403378 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:54.403475 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:54.403817 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 19:05:54.403866 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:05:54.405699 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:54.399726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43783 (local address 127.1.154.130:49667)
0620 19:05:54.400451 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:54.400502 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:54.401858 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:54.402344 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:54.402351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:54.402432 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:54.403058 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:05:54.403060 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:54.403104 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:05:54.403168 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:54.403556 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:54.403570 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:54.403579 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 19:05:54.403934 (+ 355us) client_negotiation.cc:240] Negotiation successful
0620 19:05:54.404334 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":134,"threads_started":1}
I20260620 19:05:54.406864 11703 heartbeater.cc:347] Connected to a master server at 127.1.154.190:43783
I20260620 19:05:54.407124 11703 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:54.407651 11703 heartbeater.cc:511] Master 127.1.154.190:43783 requested a full tablet report, sending...
I20260620 19:05:54.408792 11384 ts_manager.cc:194] Registered new tserver with Master: 9734afbf344848a9a16bbbd914b00a0a (127.1.154.130:39469)
W20260620 19:05:54.517688 11706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:54.517982 11706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:54.518056 11706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:54.522261 11706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:54.522362 11706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:54.522454 11706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:54.528909 11706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:54.530236 11706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:54.531772 11706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:54.542301 11714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:54.542335 11712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:54.542258 11711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:54.542580 11706 server_base.cc:1034] running on GCE node
I20260620 19:05:54.543012 11706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:54.543689 11706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:54.544895 11706 hybrid_clock.cc:648] HybridClock initialized: now 1781982354544854 us; error 62 us; skew 500 ppm
I20260620 19:05:54.546994 11706 webserver.cc:460] Webserver started at http://127.1.154.131:42585/ using document root <none> and password file <none>
I20260620 19:05:54.547720 11706 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:54.547847 11706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:54.548086 11706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:54.550387 11706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "5f9f198ae601447792ee232ecee2236c"
format_stamp: "Formatted at 2026-06-20 19:05:54 on dist-test-slave-6wrf"
server_key: "db7cde29b17fa4b93187c88566bb0e0a"
server_key_iv: "89919796790d4fea5f49932d2311b816"
server_key_version: "encryptionkey@0"
I20260620 19:05:54.550937 11706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "5f9f198ae601447792ee232ecee2236c"
format_stamp: "Formatted at 2026-06-20 19:05:54 on dist-test-slave-6wrf"
server_key: "db7cde29b17fa4b93187c88566bb0e0a"
server_key_iv: "89919796790d4fea5f49932d2311b816"
server_key_version: "encryptionkey@0"
I20260620 19:05:54.555080 11706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 19:05:54.557837 11720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:54.559099 11706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:05:54.559247 11706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "5f9f198ae601447792ee232ecee2236c"
format_stamp: "Formatted at 2026-06-20 19:05:54 on dist-test-slave-6wrf"
server_key: "db7cde29b17fa4b93187c88566bb0e0a"
server_key_iv: "89919796790d4fea5f49932d2311b816"
server_key_version: "encryptionkey@0"
I20260620 19:05:54.559386 11706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:54.565764 11706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:54.566715 11706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:54.566957 11706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:54.567694 11706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:54.569041 11706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:54.569144 11706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:54.569236 11706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:54.569288 11706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:54.580834 11833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:35311 every 8 connection(s)
I20260620 19:05:54.580947 11706 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:35311
I20260620 19:05:54.582150 11706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:54.588644 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 11706
I20260620 19:05:54.588871 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:05:54.589082 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:54.584544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:60795 (local address 127.1.154.190:43783)
0620 19:05:54.584797 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:54.584801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:54.585898 (+ 1097us) server_negotiation.cc:407] Connection header received
0620 19:05:54.587261 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:54.587264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:54.587334 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:54.587436 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:54.588573 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:54.588577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:54.588579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:05:54.588596 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:54.588624 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:05:54.588640 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:54.588759 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:54.588902 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 19:05:54.588947 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260620 19:05:54.589149 1642 external_mini_cluster.cc:1428] Setting key f156f4039b558e931bade2af4c912420
I20260620 19:05:54.590212 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:54.584965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43783 (local address 127.1.154.131:60795)
0620 19:05:54.585744 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:54.585785 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:54.586997 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:54.587600 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:54.587609 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:54.587698 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:54.588315 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:05:54.588317 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:54.588355 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:05:54.588421 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:54.588742 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:54.588750 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:54.588755 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:05:54.588944 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 19:05:54.589268 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":140,"threads_started":1}
I20260620 19:05:54.591315 11834 heartbeater.cc:347] Connected to a master server at 127.1.154.190:43783
I20260620 19:05:54.591557 11834 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:54.591992 11834 heartbeater.cc:511] Master 127.1.154.190:43783 requested a full tablet report, sending...
I20260620 19:05:54.593271 11384 ts_manager.cc:194] Registered new tserver with Master: 5f9f198ae601447792ee232ecee2236c (127.1.154.131:35311)
I20260620 19:05:54.604288 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:05:54.607916 11423 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:05:54.606697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.1.154.190:43783)
0620 19:05:54.606882 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:54.606886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:54.607267 (+ 381us) server_negotiation.cc:407] Connection header received
0620 19:05:54.607433 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:54.607438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:54.607479 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:54.607568 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:54.607803 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40578: BlockingRecv error: recv got EOF from 127.0.0.1:40578 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 19:05:54.608806 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 11444
I20260620 19:05:54.614715 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 11575
I20260620 19:05:54.619871 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 11706
I20260620 19:05:54.627128 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 11353
2026-06-20T19:05:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (881 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:05:54 dist-test-slave-6wrf krb5kdc[11844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:05:54 dist-test-slave-6wrf krb5kdc[11844](info): set up 2 sockets
Jun 20 19:05:54 dist-test-slave-6wrf krb5kdc[11844](info): commencing operation
krb5kdc: starting...
W20260620 19:05:55.694869 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:05:55 dist-test-slave-6wrf krb5kdc[11844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:05:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:05:55.826462 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:40369
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:40369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:55.939946 11857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:55.940241 11857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:55.940285 11857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:55.944367 11857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:05:55.944438 11857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:55.944459 11857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:55.944475 11857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:05:55.944499 11857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:05:55.951982 11857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:40369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:40369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:55.953542 11857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:55.954924 11857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:55.966041 11865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:55.965997 11863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:55.966048 11862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:55.966518 11857 server_base.cc:1034] running on GCE node
I20260620 19:05:55.967193 11857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:55.968547 11857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:55.969753 11857 hybrid_clock.cc:648] HybridClock initialized: now 1781982355969728 us; error 48 us; skew 500 ppm
Jun 20 19:05:55 dist-test-slave-6wrf krb5kdc[11844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982355, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:55.973379 11857 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:05:55.974870 11857 webserver.cc:460] Webserver started at http://127.1.154.190:33625/ using document root <none> and password file <none>
I20260620 19:05:55.975446 11857 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:55.975508 11857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:55.975697 11857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:55.977869 11857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "d07b0182c93448e58be96bfece3fafb6"
format_stamp: "Formatted at 2026-06-20 19:05:55 on dist-test-slave-6wrf"
server_key: "9aebabb259affe8dd564e66d4a1d7848"
server_key_iv: "d69c3ea4f3222ac0c587e96f9121003b"
server_key_version: "encryptionkey@0"
I20260620 19:05:55.978404 11857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "d07b0182c93448e58be96bfece3fafb6"
format_stamp: "Formatted at 2026-06-20 19:05:55 on dist-test-slave-6wrf"
server_key: "9aebabb259affe8dd564e66d4a1d7848"
server_key_iv: "d69c3ea4f3222ac0c587e96f9121003b"
server_key_version: "encryptionkey@0"
I20260620 19:05:55.982679 11857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:05:55.985080 11872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:55.986364 11857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:05:55.986505 11857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "d07b0182c93448e58be96bfece3fafb6"
format_stamp: "Formatted at 2026-06-20 19:05:55 on dist-test-slave-6wrf"
server_key: "9aebabb259affe8dd564e66d4a1d7848"
server_key_iv: "d69c3ea4f3222ac0c587e96f9121003b"
server_key_version: "encryptionkey@0"
I20260620 19:05:55.986620 11857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:56.019243 11857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:56.022673 11857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:56.022891 11857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:56.030189 11857 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:40369
I20260620 19:05:56.030588 11924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:40369 every 8 connection(s)
I20260620 19:05:56.031231 11857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:05:56.033903 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 11857
I20260620 19:05:56.034032 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:05:56.034300 1642 external_mini_cluster.cc:1428] Setting key b0c181987385d4a7ff4ecc4760375262
I20260620 19:05:56.034786 11925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:56.041033 11925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6: Bootstrap starting.
Jun 20 19:05:56 dist-test-slave-6wrf krb5kdc[11844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:56.043812 11925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:56.044689 11925 log.cc:824] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:56.046967 11925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6: No bootstrap required, opened a new log
I20260620 19:05:56.051054 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:56.036028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.1.154.190:40369)
0620 19:05:56.036478 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:56.036492 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:56.036528 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:05:56.037437 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:56.037453 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:56.037832 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:56.038162 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:56.038969 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.039944 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.040793 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.041321 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.041483 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:56.044199 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:56.044204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:56.044217 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:56.044244 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:56.047126 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:56.047746 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:56.047758 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:56.047765 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:56.047911 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:56.048218 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:56.048226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:56.048228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:56.048740 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:56.049251 (+ 511us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:56.049550 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 19:05:56.049994 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":139,"threads_started":1}
I20260620 19:05:56.051126 11925 raft_consensus.cc:348] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b0182c93448e58be96bfece3fafb6" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40369 } }
I20260620 19:05:56.051510 11925 raft_consensus.cc:374] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:56.051590 11925 raft_consensus.cc:729] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d07b0182c93448e58be96bfece3fafb6, State: Initialized, Role: FOLLOWER
I20260620 19:05:56.052155 11925 consensus_queue.cc:260] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b0182c93448e58be96bfece3fafb6" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40369 } }
I20260620 19:05:56.052330 11925 raft_consensus.cc:388] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:05:56.052433 11925 raft_consensus.cc:482] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:05:56.052551 11925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:56.053682 11925 raft_consensus.cc:504] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b0182c93448e58be96bfece3fafb6" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40369 } }
I20260620 19:05:56.054085 11925 leader_election.cc:302] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d07b0182c93448e58be96bfece3fafb6; no voters:
I20260620 19:05:56.054514 11925 leader_election.cc:288] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:05:56.054615 11930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:56.054862 11930 raft_consensus.cc:686] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [term 1 LEADER]: Becoming Leader. State: Replica: d07b0182c93448e58be96bfece3fafb6, State: Running, Role: LEADER
I20260620 19:05:56.055572 11930 consensus_queue.cc:237] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b0182c93448e58be96bfece3fafb6" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40369 } }
I20260620 19:05:56.056269 11925 sys_catalog.cc:564] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:05:56.057649 11932 sys_catalog.cc:455] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d07b0182c93448e58be96bfece3fafb6. Latest consensus state: current_term: 1 leader_uuid: "d07b0182c93448e58be96bfece3fafb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b0182c93448e58be96bfece3fafb6" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40369 } } }
I20260620 19:05:56.057773 11932 sys_catalog.cc:458] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [sys.catalog]: This master's current role is: LEADER
I20260620 19:05:56.058291 11931 sys_catalog.cc:455] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d07b0182c93448e58be96bfece3fafb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b0182c93448e58be96bfece3fafb6" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40369 } } }
I20260620 19:05:56.058492 11931 sys_catalog.cc:458] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6 [sys.catalog]: This master's current role is: LEADER
W20260620 19:05:56.060994 11945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:05:56.061061 11945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:05:56.061398 11939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:05:56.062319 11939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:05:56.069373 11939 catalog_manager.cc:1347] Generated new cluster ID: 10e01e1b5fb040c8a59f54c498f0e3f5
I20260620 19:05:56.069514 11939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:05:56.091169 11939 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:05:56.092449 11939 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:05:56.102635 11939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6: Generated new TSK 0
I20260620 19:05:56.103818 11939 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:05:56.138142 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40369
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:56.254590 11951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:56.254894 11951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:56.254940 11951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:56.259351 11951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:56.259426 11951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:56.259495 11951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:05:56.264994 11951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:56.266144 11951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:56.267143 11951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:56.277427 11956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:56.277428 11957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:56.277519 11951 server_base.cc:1034] running on GCE node
W20260620 19:05:56.277449 11959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:56.278388 11951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:56.279341 11951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:56.280587 11951 hybrid_clock.cc:648] HybridClock initialized: now 1781982356280573 us; error 34 us; skew 500 ppm
Jun 20 19:05:56 dist-test-slave-6wrf krb5kdc[11844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982356, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:56.283605 11951 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:05:56.284504 11951 webserver.cc:460] Webserver started at http://127.1.154.129:39877/ using document root <none> and password file <none>
I20260620 19:05:56.285054 11951 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:56.285153 11951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:56.285378 11951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:56.287417 11951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "092325de5e68417a968a000d1b57ccec"
format_stamp: "Formatted at 2026-06-20 19:05:56 on dist-test-slave-6wrf"
server_key: "0175fbb182ce9232a4b06a2f34809b9e"
server_key_iv: "a30f2f80e02633df3e4cd5cd0d665f2d"
server_key_version: "encryptionkey@0"
I20260620 19:05:56.287990 11951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "092325de5e68417a968a000d1b57ccec"
format_stamp: "Formatted at 2026-06-20 19:05:56 on dist-test-slave-6wrf"
server_key: "0175fbb182ce9232a4b06a2f34809b9e"
server_key_iv: "a30f2f80e02633df3e4cd5cd0d665f2d"
server_key_version: "encryptionkey@0"
I20260620 19:05:56.292222 11951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:05:56.294947 11966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:56.296725 11951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:05:56.296898 11951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "092325de5e68417a968a000d1b57ccec"
format_stamp: "Formatted at 2026-06-20 19:05:56 on dist-test-slave-6wrf"
server_key: "0175fbb182ce9232a4b06a2f34809b9e"
server_key_iv: "a30f2f80e02633df3e4cd5cd0d665f2d"
server_key_version: "encryptionkey@0"
I20260620 19:05:56.297019 11951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:56.313169 11951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:56.317553 11951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:56.317816 11951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:56.318449 11951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:56.319306 11951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:56.319376 11951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:56.319443 11951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:56.319484 11951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:56.328711 11951 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:45767
I20260620 19:05:56.328760 12079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:45767 every 8 connection(s)
I20260620 19:05:56.329654 11951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:05:56.336392 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 11951
I20260620 19:05:56.336535 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:05:56.336865 1642 external_mini_cluster.cc:1428] Setting key 2b5fd19ba8e4b8188e9a40051eaab1b4
Jun 20 19:05:56 dist-test-slave-6wrf krb5kdc[11844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982356, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:56.346462 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:56.331785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:50039 (local address 127.1.154.190:40369)
0620 19:05:56.332055 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:56.332060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:56.332856 (+ 796us) server_negotiation.cc:407] Connection header received
0620 19:05:56.334199 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:56.334201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:56.334253 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:56.334333 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:56.335434 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.335978 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.337774 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.338011 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.338096 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:56.341207 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:56.341211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:56.341214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:56.341248 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:56.343380 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:56.344258 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:56.344263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:56.344265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:56.344369 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:56.344800 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:56.344805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:56.344807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:56.345059 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:56.345199 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:56.346039 (+ 840us) server_negotiation.cc:299] Negotiation successful
0620 19:05:56.346235 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
I20260620 19:05:56.347621 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:56.332175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:40369 (local address 127.1.154.129:50039)
0620 19:05:56.332679 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:56.332722 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:56.333993 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:56.334461 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:56.334467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:56.334824 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:56.335299 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:56.335312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.336180 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.336186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:56.337576 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:56.337586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.338171 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.338176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:56.338336 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:56.339042 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:56.339094 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:56.341025 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:56.343627 (+ 2602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:56.343637 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:56.343671 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:56.344093 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:56.344493 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:56.344497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:56.344499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:56.344685 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:56.345243 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:56.345255 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:56.345728 (+ 473us) client_negotiation.cc:769] Sending connection context
0620 19:05:56.345994 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 19:05:56.346466 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":118,"threads_started":1}
I20260620 19:05:56.349290 12080 heartbeater.cc:347] Connected to a master server at 127.1.154.190:40369
I20260620 19:05:56.349663 12080 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
I20260620 19:05:56.350271 12080 heartbeater.cc:511] Master 127.1.154.190:40369 requested a full tablet report, sending...
I20260620 19:05:56.352210 11889 ts_manager.cc:194] Registered new tserver with Master: 092325de5e68417a968a000d1b57ccec (127.1.154.129:45767)
I20260620 19:05:56.353725 11889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:50039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:05:56.370646 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40369
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:56.488897 12085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:56.489173 12085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:56.489216 12085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:56.493830 12085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:56.493906 12085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:56.493992 12085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:05:56.499686 12085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:56.501098 12085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:56.502226 12085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:56.512395 12091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:56.512415 12090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:56.512543 12093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:56.512543 12085 server_base.cc:1034] running on GCE node
I20260620 19:05:56.513134 12085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:56.513857 12085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:56.515065 12085 hybrid_clock.cc:648] HybridClock initialized: now 1781982356515030 us; error 46 us; skew 500 ppm
Jun 20 19:05:56 dist-test-slave-6wrf krb5kdc[11844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982356, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:56.529006 12085 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:05:56.530210 12085 webserver.cc:460] Webserver started at http://127.1.154.130:37627/ using document root <none> and password file <none>
I20260620 19:05:56.530798 12085 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:56.530901 12085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:56.531142 12085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:56.533205 12085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "072edcedf99a44608e3baa607dce6d4a"
format_stamp: "Formatted at 2026-06-20 19:05:56 on dist-test-slave-6wrf"
server_key: "9893e7d7b3a2d8ffbd4a7100b019bc87"
server_key_iv: "123cc69637b8158374d5512e1ddf8373"
server_key_version: "encryptionkey@0"
I20260620 19:05:56.533814 12085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "072edcedf99a44608e3baa607dce6d4a"
format_stamp: "Formatted at 2026-06-20 19:05:56 on dist-test-slave-6wrf"
server_key: "9893e7d7b3a2d8ffbd4a7100b019bc87"
server_key_iv: "123cc69637b8158374d5512e1ddf8373"
server_key_version: "encryptionkey@0"
I20260620 19:05:56.537485 12085 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:05:56.540001 12100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:56.541774 12085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 19:05:56.541925 12085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "072edcedf99a44608e3baa607dce6d4a"
format_stamp: "Formatted at 2026-06-20 19:05:56 on dist-test-slave-6wrf"
server_key: "9893e7d7b3a2d8ffbd4a7100b019bc87"
server_key_iv: "123cc69637b8158374d5512e1ddf8373"
server_key_version: "encryptionkey@0"
I20260620 19:05:56.542043 12085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:56.582927 12085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:56.586198 12085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:56.586440 12085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:56.587059 12085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:56.587913 12085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:56.587984 12085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:56.588053 12085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:56.588104 12085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:56.597369 12085 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:46077
I20260620 19:05:56.597391 12213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:46077 every 8 connection(s)
I20260620 19:05:56.598310 12085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:05:56.598910 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 12085
I20260620 19:05:56.599020 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:05:56.599298 1642 external_mini_cluster.cc:1428] Setting key b2b9cdfd9988f2d597605b2a9a3396ad
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
Jun 20 19:05:56 dist-test-slave-6wrf krb5kdc[11844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982356, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:56.620060 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:56.600341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:39155 (local address 127.1.154.190:40369)
0620 19:05:56.600515 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:56.600518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:56.601580 (+ 1062us) server_negotiation.cc:407] Connection header received
0620 19:05:56.603602 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:56.603607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:56.603670 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:56.604057 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:56.605423 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.606353 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.607976 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.608445 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.608569 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:56.614520 (+ 5951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:56.614525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:56.614528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:56.614573 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:56.616702 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:56.617376 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:56.617384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:56.617388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:56.617511 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:56.617996 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:56.618003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:56.618008 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:56.618454 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:56.618609 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:56.619540 (+ 931us) server_negotiation.cc:299] Negotiation successful
0620 19:05:56.619747 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 19:05:56.620901 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:56.600846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:40369 (local address 127.1.154.130:39155)
0620 19:05:56.601418 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:56.601480 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:56.603363 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:56.604244 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:56.604251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:56.604629 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:56.605237 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:56.605251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.606615 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.606626 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:56.607772 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:56.607792 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.608649 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.608654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:56.608835 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:56.609507 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:56.609525 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:56.614159 (+ 4634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:56.616908 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:56.616914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:56.616929 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:56.617225 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:56.617692 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:56.617695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:56.617696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:56.617836 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:56.618738 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:56.618748 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:56.619162 (+ 414us) client_negotiation.cc:769] Sending connection context
0620 19:05:56.619406 (+ 244us) client_negotiation.cc:240] Negotiation successful
0620 19:05:56.619882 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":161,"threads_started":1}
I20260620 19:05:56.622551 12214 heartbeater.cc:347] Connected to a master server at 127.1.154.190:40369
I20260620 19:05:56.623003 12214 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:56.623664 12214 heartbeater.cc:511] Master 127.1.154.190:40369 requested a full tablet report, sending...
I20260620 19:05:56.624902 11889 ts_manager.cc:194] Registered new tserver with Master: 072edcedf99a44608e3baa607dce6d4a (127.1.154.130:46077)
I20260620 19:05:56.625589 11889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:39155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:05:56.633985 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40369
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:05:56.748241 12219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:05:56.748570 12219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:05:56.748651 12219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:05:56.752800 12219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:05:56.752908 12219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:05:56.753003 12219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:05:56.758909 12219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:05:56.760635 12219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:05:56.762104 12219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:05:56.774705 12224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:56.774803 12227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:05:56.774905 12225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:05:56.775277 12219 server_base.cc:1034] running on GCE node
I20260620 19:05:56.775748 12219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:05:56.776436 12219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:05:56.777806 12219 hybrid_clock.cc:648] HybridClock initialized: now 1781982356777798 us; error 45 us; skew 500 ppm
Jun 20 19:05:56 dist-test-slave-6wrf krb5kdc[11844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982356, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:05:56.780781 12219 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:05:56.781806 12219 webserver.cc:460] Webserver started at http://127.1.154.131:34777/ using document root <none> and password file <none>
I20260620 19:05:56.782936 12219 fs_manager.cc:359] Metadata directory not provided
I20260620 19:05:56.783070 12219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:05:56.783330 12219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:05:56.785382 12219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "ba02903e30a547b2a9420e6012fa4700"
format_stamp: "Formatted at 2026-06-20 19:05:56 on dist-test-slave-6wrf"
server_key: "3d1bc24fb0257f30008758dcedddb5d0"
server_key_iv: "2df1ccf98de637e0eec446a11c180461"
server_key_version: "encryptionkey@0"
I20260620 19:05:56.786043 12219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "ba02903e30a547b2a9420e6012fa4700"
format_stamp: "Formatted at 2026-06-20 19:05:56 on dist-test-slave-6wrf"
server_key: "3d1bc24fb0257f30008758dcedddb5d0"
server_key_iv: "2df1ccf98de637e0eec446a11c180461"
server_key_version: "encryptionkey@0"
I20260620 19:05:56.789903 12219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:05:56.792870 12234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:56.795027 12219 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:05:56.795200 12219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "ba02903e30a547b2a9420e6012fa4700"
format_stamp: "Formatted at 2026-06-20 19:05:56 on dist-test-slave-6wrf"
server_key: "3d1bc24fb0257f30008758dcedddb5d0"
server_key_iv: "2df1ccf98de637e0eec446a11c180461"
server_key_version: "encryptionkey@0"
I20260620 19:05:56.795329 12219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:05:56.823576 12219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:05:56.827910 12219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:05:56.828249 12219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:05:56.828893 12219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:05:56.829846 12219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:05:56.829918 12219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:56.829998 12219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:05:56.830049 12219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:05:56.840262 12219 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:45153
I20260620 19:05:56.840286 12347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:45153 every 8 connection(s)
I20260620 19:05:56.841316 12219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:05:56.842339 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 12219
I20260620 19:05:56.842463 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:05:56.842767 1642 external_mini_cluster.cc:1428] Setting key 1731e8659a0f551a2aad72f6c7f79ffa
Jun 20 19:05:56 dist-test-slave-6wrf krb5kdc[11844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982356, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:05:56.860615 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:56.843608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:36961 (local address 127.1.154.190:40369)
0620 19:05:56.843749 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:56.843754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:56.845068 (+ 1314us) server_negotiation.cc:407] Connection header received
0620 19:05:56.847087 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:56.847090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:56.847139 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:56.847222 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:56.848965 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.849772 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.851163 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.851381 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.851479 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:56.855143 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:56.855148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:56.855151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:56.855195 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:56.857676 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:56.858550 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:56.858557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:56.858560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:56.858708 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:56.859153 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:56.859157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:56.859159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:56.859421 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:56.859547 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:56.860242 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 19:05:56.860412 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 19:05:56.862082 12350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:56.844090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:40369 (local address 127.1.154.131:36961)
0620 19:05:56.844829 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:56.844925 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:56.846816 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:56.847462 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:56.847471 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:56.848073 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:05:56.848821 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:56.848849 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.850048 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.850052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:56.851031 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:56.851042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.851556 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.851560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:56.851911 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:56.852798 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:05:56.852819 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:05:56.854865 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:05:56.857846 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:56.857856 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:56.857883 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:56.858417 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:56.858818 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:05:56.858822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:05:56.858824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:05:56.859026 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:05:56.859546 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:05:56.859556 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:05:56.859933 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 19:05:56.860180 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 19:05:56.860645 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":104,"threads_started":1}
I20260620 19:05:56.863516 12348 heartbeater.cc:347] Connected to a master server at 127.1.154.190:40369
I20260620 19:05:56.863945 12348 heartbeater.cc:464] Registering TS with master...
I20260620 19:05:56.864487 12348 heartbeater.cc:511] Master 127.1.154.190:40369 requested a full tablet report, sending...
I20260620 19:05:56.865554 11889 ts_manager.cc:194] Registered new tserver with Master: ba02903e30a547b2a9420e6012fa4700 (127.1.154.131:45153)
I20260620 19:05:56.866199 11889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:36961
I20260620 19:05:56.870896 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:05:56.882035 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:56.873125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45070 (local address 127.1.154.190:40369)
0620 19:05:56.873302 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:56.873307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:56.873449 (+ 142us) server_negotiation.cc:407] Connection header received
0620 19:05:56.873633 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:56.873636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:56.873686 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:56.873777 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:05:56.874500 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.875017 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.875848 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.876072 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.876185 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:56.878182 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:05:56.878186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:05:56.878189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:05:56.878234 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:05:56.879962 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:56.880528 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:56.880534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:56.880536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:56.880629 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:05:56.880973 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:05:56.880976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:05:56.880978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:05:56.881231 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:05:56.881388 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:56.881706 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 19:05:56.881844 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 19:05:56.885262 11889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:05:56.888955 11889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:05:56.905346 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:56.899380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:45767 (local address 127.0.0.1:47122)
0620 19:05:56.900578 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:56.900591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:56.900698 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:56.901253 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:56.901259 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:56.901283 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:56.901428 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:56.901435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.902555 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.902558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:56.903908 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:56.903918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.904806 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.904810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:56.904917 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:56.904954 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 19:05:56.905057 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 19:05:56.905186 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"mutex_wait_us":151,"thread_start_us":91,"threads_started":1}
I20260620 19:05:56.905368 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:56.899996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47122 (local address 127.1.154.129:45767)
0620 19:05:56.900403 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:56.900410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:56.900683 (+ 273us) server_negotiation.cc:407] Connection header received
0620 19:05:56.900838 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:56.900843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:56.901073 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:56.901242 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:56.901562 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.902424 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.904059 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.904670 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.904791 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:56.904955 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:56.905097 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 19:05:56.905197 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":91,"threads_started":1}
I20260620 19:05:56.906067 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:56.899432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:45153 (local address 127.0.0.1:49844)
0620 19:05:56.900520 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:56.900533 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:56.900613 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:56.901427 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:56.901430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:56.901459 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:56.901582 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:56.901639 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.903077 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.903083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:56.904508 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:56.904518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.905587 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.905615 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:56.905714 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:56.905739 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 19:05:56.905842 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 19:05:56.905918 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":39,"spinlock_wait_cycles":896,"thread_start_us":117,"threads_started":1}
I20260620 19:05:56.906250 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:56.900167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.1.154.131:45153)
0620 19:05:56.900957 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:56.900967 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:56.900986 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:05:56.901053 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:56.901057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:56.901267 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:56.901438 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:56.901749 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.902912 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.904656 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.905474 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.905588 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:56.905753 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:56.905868 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 19:05:56.905956 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":140,"threads_started":1}
I20260620 19:05:56.906977 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:56.899380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:46077 (local address 127.0.0.1:33758)
0620 19:05:56.900296 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:56.900349 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:56.900488 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:56.901409 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:56.901411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:56.901425 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:56.901726 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:56.901733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.903108 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.903110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:56.905093 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:56.905100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.906560 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.906564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:56.906655 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:56.906684 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 19:05:56.906758 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:05:56.906840 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":321,"spinlock_wait_cycles":4352,"thread_start_us":87,"threads_started":1}
I20260620 19:05:56.907405 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:56.899637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33758 (local address 127.1.154.130:46077)
0620 19:05:56.900861 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:56.900872 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:56.900894 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:05:56.900967 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:56.900971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:56.901134 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:56.901284 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:56.901850 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.902961 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.905215 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:56.906275 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:56.906382 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:56.906498 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:56.907163 (+ 665us) server_negotiation.cc:299] Negotiation successful
0620 19:05:56.907254 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1128,"thread_start_us":117,"threads_started":1}
I20260620 19:05:56.909018 12282 tablet_service.cc:1467] Processing CreateTablet for tablet d6dc838a705244e8896b8cbc9ec63472 (DEFAULT_TABLE table=test-table [id=b7965a33a0114d1ca561aabf2feacaba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:56.909009 12014 tablet_service.cc:1467] Processing CreateTablet for tablet d6dc838a705244e8896b8cbc9ec63472 (DEFAULT_TABLE table=test-table [id=b7965a33a0114d1ca561aabf2feacaba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:56.910022 12282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6dc838a705244e8896b8cbc9ec63472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:56.910051 12014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6dc838a705244e8896b8cbc9ec63472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:56.909806 12148 tablet_service.cc:1467] Processing CreateTablet for tablet d6dc838a705244e8896b8cbc9ec63472 (DEFAULT_TABLE table=test-table [id=b7965a33a0114d1ca561aabf2feacaba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:05:56.910696 12148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6dc838a705244e8896b8cbc9ec63472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:05:56.917026 12365 tablet_bootstrap.cc:492] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700: Bootstrap starting.
I20260620 19:05:56.918352 12366 tablet_bootstrap.cc:492] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec: Bootstrap starting.
I20260620 19:05:56.919380 12365 tablet_bootstrap.cc:654] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:56.920686 12366 tablet_bootstrap.cc:654] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:56.920696 12365 log.cc:824] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:56.921394 12367 tablet_bootstrap.cc:492] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a: Bootstrap starting.
I20260620 19:05:56.921471 12366 log.cc:824] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:56.922796 12365 tablet_bootstrap.cc:492] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700: No bootstrap required, opened a new log
I20260620 19:05:56.922937 12365 ts_tablet_manager.cc:1397] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:05:56.923413 12366 tablet_bootstrap.cc:492] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec: No bootstrap required, opened a new log
I20260620 19:05:56.923583 12366 ts_tablet_manager.cc:1397] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:05:56.924243 12367 tablet_bootstrap.cc:654] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a: Neither blocks nor log segments found. Creating new log.
I20260620 19:05:56.925014 12367 log.cc:824] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:05:56.925782 12365 raft_consensus.cc:348] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072edcedf99a44608e3baa607dce6d4a" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46077 } } peers { permanent_uuid: "092325de5e68417a968a000d1b57ccec" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45767 } } peers { permanent_uuid: "ba02903e30a547b2a9420e6012fa4700" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45153 } }
I20260620 19:05:56.926059 12365 raft_consensus.cc:374] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:56.926110 12365 raft_consensus.cc:729] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba02903e30a547b2a9420e6012fa4700, State: Initialized, Role: FOLLOWER
I20260620 19:05:56.926550 12365 consensus_queue.cc:260] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072edcedf99a44608e3baa607dce6d4a" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46077 } } peers { permanent_uuid: "092325de5e68417a968a000d1b57ccec" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45767 } } peers { permanent_uuid: "ba02903e30a547b2a9420e6012fa4700" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45153 } }
I20260620 19:05:56.927011 12367 tablet_bootstrap.cc:492] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a: No bootstrap required, opened a new log
I20260620 19:05:56.927146 12348 heartbeater.cc:503] Master 127.1.154.190:40369 was elected leader, sending a full tablet report...
I20260620 19:05:56.927179 12367 ts_tablet_manager.cc:1397] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:05:56.927488 12365 ts_tablet_manager.cc:1428] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:05:56.927960 12366 raft_consensus.cc:348] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072edcedf99a44608e3baa607dce6d4a" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46077 } } peers { permanent_uuid: "092325de5e68417a968a000d1b57ccec" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45767 } } peers { permanent_uuid: "ba02903e30a547b2a9420e6012fa4700" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45153 } }
I20260620 19:05:56.928191 12366 raft_consensus.cc:374] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:56.928236 12366 raft_consensus.cc:729] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092325de5e68417a968a000d1b57ccec, State: Initialized, Role: FOLLOWER
I20260620 19:05:56.928660 12366 consensus_queue.cc:260] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072edcedf99a44608e3baa607dce6d4a" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46077 } } peers { permanent_uuid: "092325de5e68417a968a000d1b57ccec" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45767 } } peers { permanent_uuid: "ba02903e30a547b2a9420e6012fa4700" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45153 } }
I20260620 19:05:56.929557 12366 ts_tablet_manager.cc:1428] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:05:56.929916 12080 heartbeater.cc:503] Master 127.1.154.190:40369 was elected leader, sending a full tablet report...
I20260620 19:05:56.932003 12367 raft_consensus.cc:348] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072edcedf99a44608e3baa607dce6d4a" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46077 } } peers { permanent_uuid: "092325de5e68417a968a000d1b57ccec" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45767 } } peers { permanent_uuid: "ba02903e30a547b2a9420e6012fa4700" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45153 } }
I20260620 19:05:56.932718 12367 raft_consensus.cc:374] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:05:56.932788 12367 raft_consensus.cc:729] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 072edcedf99a44608e3baa607dce6d4a, State: Initialized, Role: FOLLOWER
I20260620 19:05:56.933223 12367 consensus_queue.cc:260] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072edcedf99a44608e3baa607dce6d4a" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46077 } } peers { permanent_uuid: "092325de5e68417a968a000d1b57ccec" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45767 } } peers { permanent_uuid: "ba02903e30a547b2a9420e6012fa4700" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45153 } }
I20260620 19:05:56.933894 12214 heartbeater.cc:503] Master 127.1.154.190:40369 was elected leader, sending a full tablet report...
I20260620 19:05:56.934158 12367 ts_tablet_manager.cc:1428] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260620 19:05:57.081115 12081 tablet.cc:2369] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:05:57.093773 12349 tablet.cc:2369] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:05:57.099952 12215 tablet.cc:2369] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:05:57.176442 12373 raft_consensus.cc:482] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:05:57.176745 12373 raft_consensus.cc:504] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072edcedf99a44608e3baa607dce6d4a" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46077 } } peers { permanent_uuid: "092325de5e68417a968a000d1b57ccec" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45767 } } peers { permanent_uuid: "ba02903e30a547b2a9420e6012fa4700" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45153 } }
I20260620 19:05:57.177989 12373 leader_election.cc:288] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 092325de5e68417a968a000d1b57ccec (127.1.154.129:45767), ba02903e30a547b2a9420e6012fa4700 (127.1.154.131:45153)
I20260620 19:05:57.184231 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:57.178615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:58861 (local address 127.1.154.131:45153)
0620 19:05:57.178882 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:57.178889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:57.179169 (+ 280us) server_negotiation.cc:407] Connection header received
0620 19:05:57.179316 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:57.179320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:57.179372 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:57.179458 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:57.179977 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:57.180751 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:57.182771 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:57.183465 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:57.183627 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:57.183683 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:57.184026 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 19:05:57.184106 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260620 19:05:57.184231 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:57.178593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:53131 (local address 127.1.154.129:45767)
0620 19:05:57.178796 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:57.178801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:57.179088 (+ 287us) server_negotiation.cc:407] Connection header received
0620 19:05:57.179172 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:57.179175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:57.179235 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:57.179341 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:05:57.179975 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:57.180775 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:57.182771 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:57.183487 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:57.183627 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:57.183683 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:57.184026 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 19:05:57.184107 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 19:05:57.184252 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:57.178449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:45153 (local address 127.1.154.130:58861)
0620 19:05:57.179038 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:57.179057 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:57.179202 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:57.179496 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:57.179498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:57.179542 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:57.179797 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:57.179804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:57.180919 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:57.180923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:57.182591 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:57.182599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:57.183640 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:57.183644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:57.183806 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:57.183862 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 19:05:57.183949 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 19:05:57.184054 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":389,"spinlock_wait_cycles":12800,"thread_start_us":105,"threads_started":1}
I20260620 19:05:57.184254 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:57.178449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:45767 (local address 127.1.154.130:53131)
0620 19:05:57.178829 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:57.178847 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:05:57.178995 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:05:57.179488 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:05:57.179492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:05:57.179541 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:05:57.179802 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:57.179807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:57.180919 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:57.180923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:57.182588 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:05:57.182599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:05:57.183640 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:05:57.183644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:05:57.183800 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:57.183869 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 19:05:57.183949 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 19:05:57.184054 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":279,"spinlock_wait_cycles":30848,"thread_start_us":226,"threads_started":1}
I20260620 19:05:57.184998 12302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6dc838a705244e8896b8cbc9ec63472" candidate_uuid: "072edcedf99a44608e3baa607dce6d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba02903e30a547b2a9420e6012fa4700" is_pre_election: true
I20260620 19:05:57.185366 12302 raft_consensus.cc:2454] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 072edcedf99a44608e3baa607dce6d4a in term 0.
I20260620 19:05:57.185367 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6dc838a705244e8896b8cbc9ec63472" candidate_uuid: "072edcedf99a44608e3baa607dce6d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "092325de5e68417a968a000d1b57ccec" is_pre_election: true
I20260620 19:05:57.185707 12034 raft_consensus.cc:2454] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 072edcedf99a44608e3baa607dce6d4a in term 0.
I20260620 19:05:57.185887 12103 leader_election.cc:302] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 072edcedf99a44608e3baa607dce6d4a, ba02903e30a547b2a9420e6012fa4700; no voters:
I20260620 19:05:57.186190 12373 raft_consensus.cc:2781] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:05:57.186267 12373 raft_consensus.cc:482] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:05:57.186298 12373 raft_consensus.cc:3037] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:57.187361 12373 raft_consensus.cc:504] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072edcedf99a44608e3baa607dce6d4a" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46077 } } peers { permanent_uuid: "092325de5e68417a968a000d1b57ccec" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45767 } } peers { permanent_uuid: "ba02903e30a547b2a9420e6012fa4700" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45153 } }
I20260620 19:05:57.187767 12373 leader_election.cc:288] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [CANDIDATE]: Term 1 election: Requested vote from peers 092325de5e68417a968a000d1b57ccec (127.1.154.129:45767), ba02903e30a547b2a9420e6012fa4700 (127.1.154.131:45153)
I20260620 19:05:57.188123 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6dc838a705244e8896b8cbc9ec63472" candidate_uuid: "072edcedf99a44608e3baa607dce6d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "092325de5e68417a968a000d1b57ccec"
I20260620 19:05:57.188201 12302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6dc838a705244e8896b8cbc9ec63472" candidate_uuid: "072edcedf99a44608e3baa607dce6d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba02903e30a547b2a9420e6012fa4700"
I20260620 19:05:57.188237 12034 raft_consensus.cc:3037] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:57.188370 12302 raft_consensus.cc:3037] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:05:57.189393 12034 raft_consensus.cc:2454] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 072edcedf99a44608e3baa607dce6d4a in term 1.
I20260620 19:05:57.189615 12302 raft_consensus.cc:2454] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 072edcedf99a44608e3baa607dce6d4a in term 1.
I20260620 19:05:57.189757 12102 leader_election.cc:302] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 072edcedf99a44608e3baa607dce6d4a, 092325de5e68417a968a000d1b57ccec; no voters:
I20260620 19:05:57.189944 12373 raft_consensus.cc:2781] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:05:57.190240 12373 raft_consensus.cc:686] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [term 1 LEADER]: Becoming Leader. State: Replica: 072edcedf99a44608e3baa607dce6d4a, State: Running, Role: LEADER
I20260620 19:05:57.190699 12373 consensus_queue.cc:237] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072edcedf99a44608e3baa607dce6d4a" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46077 } } peers { permanent_uuid: "092325de5e68417a968a000d1b57ccec" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45767 } } peers { permanent_uuid: "ba02903e30a547b2a9420e6012fa4700" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45153 } }
I20260620 19:05:57.194129 11888 catalog_manager.cc:5495] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 072edcedf99a44608e3baa607dce6d4a (127.1.154.130). New cstate: current_term: 1 leader_uuid: "072edcedf99a44608e3baa607dce6d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072edcedf99a44608e3baa607dce6d4a" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 46077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "092325de5e68417a968a000d1b57ccec" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba02903e30a547b2a9420e6012fa4700" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45153 } health_report { overall_health: UNKNOWN } } }
I20260620 19:05:57.272541 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:57.268961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.1.154.130:46077)
0620 19:05:57.269119 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:05:57.269123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:05:57.269140 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:05:57.269236 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:05:57.269238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:05:57.269291 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:05:57.269419 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:05:57.269702 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:57.270197 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:57.271065 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:05:57.271280 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:05:57.271402 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:05:57.271706 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:05:57.272108 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:05:57.272238 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 19:05:57.272298 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 19:05:57.272402 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 19:05:57.278482 12034 raft_consensus.cc:1261] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec [term 1 FOLLOWER]: Refusing update from remote peer 072edcedf99a44608e3baa607dce6d4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:57.278463 12302 raft_consensus.cc:1261] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700 [term 1 FOLLOWER]: Refusing update from remote peer 072edcedf99a44608e3baa607dce6d4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:05:57.279156 12376 consensus_queue.cc:1035] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba02903e30a547b2a9420e6012fa4700" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:57.279330 12373 consensus_queue.cc:1035] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "092325de5e68417a968a000d1b57ccec" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:05:57.287321 12380 mvcc.cc:205] Tried to move back new op lower bound from 7298999735405301760 to 7298999735055884288. Current Snapshot: MvccSnapshot[applied={T|T < 7298999735405301760}]
I20260620 19:05:57.298965 11887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:57.299204 11887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:05:57.301824 11887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d07b0182c93448e58be96bfece3fafb6: Sending DeleteTablet for 3 replicas of tablet d6dc838a705244e8896b8cbc9ec63472
I20260620 19:05:57.302675 12282 tablet_service.cc:1514] Processing DeleteTablet for tablet d6dc838a705244e8896b8cbc9ec63472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:57 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:49844
I20260620 19:05:57.302721 12148 tablet_service.cc:1514] Processing DeleteTablet for tablet d6dc838a705244e8896b8cbc9ec63472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:57 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:33758
I20260620 19:05:57.303180 12390 tablet_replica.cc:330] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700: stopping tablet replica
I20260620 19:05:57.303215 12014 tablet_service.cc:1514] Processing DeleteTablet for tablet d6dc838a705244e8896b8cbc9ec63472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:05:57 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:47122
I20260620 19:05:57.303556 12390 raft_consensus.cc:2229] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:57.303710 12392 tablet_replica.cc:330] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec: stopping tablet replica
I20260620 19:05:57.303889 12390 raft_consensus.cc:2258] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:57.304176 12392 raft_consensus.cc:2229] T d6dc838a705244e8896b8cbc9ec63472 P 092325de5e68417a968a000d1b57ccec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:05:57.304392 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 11951
I20260620 19:05:57.307333 12390 ts_tablet_manager.cc:1905] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:05:57.309708 12391 tablet_replica.cc:330] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a: stopping tablet replica
I20260620 19:05:57.310102 12391 raft_consensus.cc:2229] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:05:57.310602 12391 raft_consensus.cc:2258] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:05:57.310899 12390 ts_tablet_manager.cc:1918] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:05:57.310995 12390 log.cc:1196] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/d6dc838a705244e8896b8cbc9ec63472
I20260620 19:05:57.311300 12390 ts_tablet_manager.cc:1939] T d6dc838a705244e8896b8cbc9ec63472 P ba02903e30a547b2a9420e6012fa4700: Deleting consensus metadata
I20260620 19:05:57.312286 12391 ts_tablet_manager.cc:1905] T d6dc838a705244e8896b8cbc9ec63472 P 072edcedf99a44608e3baa607dce6d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:05:57.312414 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:45767 (error 108)
I20260620 19:05:57.312990 11875 catalog_manager.cc:4841] TS ba02903e30a547b2a9420e6012fa4700 (127.1.154.131:45153): tablet d6dc838a705244e8896b8cbc9ec63472 (table test-table [id=b7965a33a0114d1ca561aabf2feacaba]) successfully deleted
I20260620 19:05:57.314296 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:57.313914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:45767 (local address 127.0.0.1:47134)
0620 19:05:57.314090 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:57.314189 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 19:05:57.314286 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 12085
W20260620 19:05:57.315953 11874 catalog_manager.cc:4584] TS 092325de5e68417a968a000d1b57ccec (127.1.154.129:45767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6dc838a705244e8896b8cbc9ec63472: Network error: Client connection negotiation failed: client connection to 127.1.154.129:45767: connect: Connection refused (error 111)
I20260620 19:05:57.319952 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 12219
I20260620 19:05:57.321115 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:05:57.320710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:46077 (local address 127.0.0.1:33778)
0620 19:05:57.320924 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 19:05:57.321000 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.130:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 19:05:57.326010 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 11857
2026-06-20T19:05:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2699 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:05:57 dist-test-slave-6wrf krb5kdc[12396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:05:57 dist-test-slave-6wrf krb5kdc[12396](info): set up 2 sockets
Jun 20 19:05:57 dist-test-slave-6wrf krb5kdc[12396](info): commencing operation
krb5kdc: starting...
W20260620 19:05:58.402602 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:05:58 dist-test-slave-6wrf krb5kdc[12396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:05:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:05:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.154.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.154.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/rangeradmin_127.1.154.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/rangeradmin_127.1.154.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.154.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.154.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/rangerlookup_127.1.154.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/rangerlookup_127.1.154.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.154.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.154.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/HTTP_127.1.154.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/HTTP_127.1.154.148@KRBTEST.COM.keytab.
I20260620 19:05:58.581529 1642 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8BnTdh/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/postgres -l logfile start
2026-06-20 19:06:01.481 UTC [12426] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 19:06:01.481 UTC [12426] LOG: listening on IPv4 address "127.1.154.148", port 46217
2026-06-20 19:06:01.486 UTC [12426] LOG: listening on Unix socket "/tmp/.s.PGSQL.46217"
2026-06-20 19:06:01.521 UTC [12429] LOG: database system was shut down at 2026-06-20 19:05:59 UTC
2026-06-20 19:06:01.527 UTC [12426] LOG: database system is ready to accept connections
I20260620 19:06:02.502516 1642 mini_postgres.cc:96] Postgres bound to 46217
2026-06-20 19:06:02.509 UTC [12437] FATAL: database "slave" does not exist
127.1.154.148:46217 - accepting connections
I20260620 19:06:02.510756 1642 mini_ranger.cc:161] Starting Ranger...
I20260620 19:06:02.528210 1642 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 19:06:03.129081 1642 mini_ranger.cc:87] Created ranger Postgres database
I20260620 19:06:03.129348 1642 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-admin
2026-06-20 19:06:03,424 [I] DB FLAVOR :POSTGRES
2026-06-20 19:06:03,425 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:06:03,425 [I] Checking connection..
2026-06-20 19:06:03,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 19:06:03,685 [I] Checking connection passed.
2026-06-20 19:06:03,685 [I] --------- Verifying version history table ---------
2026-06-20 19:06:03,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:06:03,976 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:06:03,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:06:04,251 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:06:04,251 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 19:06:04,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 19:06:04.506 UTC [12501] WARNING: there is no transaction in progress
2026-06-20 19:06:04,513 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 19:06:04,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:06:04,762 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 19:06:04,763 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:06:04,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 19:06:05,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 19:06:05,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6wrf.c.gcp-upstream.internal','N') ;"
2026-06-20 19:06:05,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:06:05,828 [I] Table x_portal_user does not exist in database ranger
2026-06-20 19:06:05,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:06:06,109 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 19:06:06,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:06:06,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:06:06,624 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 19:06:06,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 19:06:07.795 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:07.814 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:07.834 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:07.853 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:07.872 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:07.893 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:07.961 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:07.969 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:07.987 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:07.999 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:08.013 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:08.038 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:08.053 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:08.062 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:08.072 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:08.082 UTC [12609] WARNING: there is no transaction in progress
2026-06-20 19:06:08,946 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 19:06:08,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 19:06:09,217 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 19:06:09,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:06:09,484 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 19:06:09,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:06:09,739 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 19:06:09,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:06:10,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:06:10,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6wrf.c.gcp-upstream.internal';"
2026-06-20 19:06:10,519 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 19:06:10,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 19:06:10,802 [I] DB_PATCHES have already been applied
I20260620 19:06:10.808831 1642 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-admin:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8BnTdh/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 19:06:10.808935 1642 mini_ranger.cc:193] Using host: 127.1.154.148
I20260620 19:06:10.812887 1642 mini_ranger.cc:227] Ranger admin URL: http://127.1.154.148:55485
Jun 20, 2026 7:06:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:06:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:06:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:06:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:06:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:06:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:06:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 19:06:11.952947 1642 mini_ranger.cc:160] Time spent starting Ranger: real 9.442s user 0.001s sys 0.011s
Jun 20, 2026 7:06:35 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:06:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Jun 20, 2026 7:06:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:06:35 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:06:35 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:06:36 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 19:06:36.812546 1642 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.154.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.154.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/rangerkms_127.1.154.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/rangerkms_127.1.154.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.154.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.154.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.154.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/HTTP_127.1.154.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.154.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/HTTP_127.1.154.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 19:06:36 dist-test-slave-6wrf krb5kdc[12396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982396, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 19:06:36.911515 1642 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 19:06:36.922729 1642 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 19:06:38.146463 1642 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 19:06:38.146548 1642 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-kms
I20260620 19:06:38.146606 1642 mini_ranger_kms.cc:227] Using postgres at 127.1.154.148:46217
2026-06-20 19:06:38,382 [I] DB FLAVOR :POSTGRES
2026-06-20 19:06:38,382 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:06:38,382 [I] Checking connection
2026-06-20 19:06:38,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 19:06:38,668 [I] connection success
2026-06-20 19:06:38,668 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 19:06:38,668 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 19:06:38,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 19:06:38,944 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 19:06:38,945 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:06:38,945 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 19:06:38,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:46217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 19:06:39,251 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 19:06:39.881084 1642 mini_ranger_kms.cc:310] Created kms service
I20260620 19:06:40.095080 1642 mini_ranger_kms.cc:326] Created kudu user
I20260620 19:06:40.138391 1642 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 19:06:40 dist-test-slave-6wrf krb5kdc[12396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982396, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
I20260620 19:06:40.644357 1642 mini_ranger_kms.cc:384] Added ranger policy
I20260620 19:06:40.644523 1642 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-kms:/tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 19:06:40.644548 1642 mini_ranger_kms.cc:240] Using host: 127.1.154.148
I20260620 19:06:40.647964 1642 mini_ranger_kms.cc:276] Ranger KMS PID: 12790
I20260620 19:06:40.648102 1642 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.154.148:46969
Jun 20, 2026 7:06:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:06:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:06:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:06:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:06:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:06:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 19:06:43.199231 1642 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.288s user 0.000s sys 0.213s
I20260620 19:06:43.199376 1642 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 19:06:43.199492 1642 mini_ranger_kms.cc:401] 127.1.154.148:46969/kms/v1/keys
Jun 20 19:06:43 dist-test-slave-6wrf krb5kdc[12396](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982403, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.154.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 19:06:43 dist-test-slave-6wrf krb5kdc[12396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982403, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.154.148@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
Jun 20 19:06:44 dist-test-slave-6wrf krb5kdc[12396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982403, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.154.148@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
Jun 20 19:06:46 dist-test-slave-6wrf krb5kdc[12396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:06:46.641003 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:46053
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:46053
--ranger_config_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.154.148:46969/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:06:46.759537 12840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:06:46.760039 12840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:06:46.760092 12840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:06:46.765960 12840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:06:46.766047 12840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:06:46.766067 12840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:06:46.766084 12840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:06:46.766099 12840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:06:46.774268 12840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.154.148:46969/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:46053
--ranger_config_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:46053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:06:46.776057 12840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:06:46.777330 12840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:06:46.788142 12848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:06:46.788245 12840 server_base.cc:1034] running on GCE node
W20260620 19:06:46.788100 12845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:06:46.788091 12846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:06:46.788973 12840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:06:46.790242 12840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:06:46.791476 12840 hybrid_clock.cc:648] HybridClock initialized: now 1781982406791457 us; error 38 us; skew 500 ppm
Jun 20 19:06:46 dist-test-slave-6wrf krb5kdc[12396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982406, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:06:46.806782 12840 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:06:46.808015 12840 webserver.cc:460] Webserver started at http://127.1.154.190:46309/ using document root <none> and password file <none>
I20260620 19:06:46.809165 12840 fs_manager.cc:359] Metadata directory not provided
I20260620 19:06:46.809299 12840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:06:46.809653 12840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:06:46 dist-test-slave-6wrf krb5kdc[12396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982406, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
I20260620 19:06:46.861155 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "98e8a2b2ad3e49108b2250218cdee599"
format_stamp: "Formatted at 2026-06-20 19:06:46 on dist-test-slave-6wrf"
server_key: "4bed7bbd78332749a58297aabcef5bd5"
server_key_iv: "4cf6d29b3ca75cd73bad96cf65525a1d"
server_key_version: "kuduclusterkey@0"
I20260620 19:06:46.861939 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "98e8a2b2ad3e49108b2250218cdee599"
format_stamp: "Formatted at 2026-06-20 19:06:46 on dist-test-slave-6wrf"
server_key: "4bed7bbd78332749a58297aabcef5bd5"
server_key_iv: "4cf6d29b3ca75cd73bad96cf65525a1d"
server_key_version: "kuduclusterkey@0"
I20260620 19:06:46.867153 12840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 19:06:46.885363 12855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:06:46.887108 12840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:06:46.887264 12840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "98e8a2b2ad3e49108b2250218cdee599"
format_stamp: "Formatted at 2026-06-20 19:06:46 on dist-test-slave-6wrf"
server_key: "4bed7bbd78332749a58297aabcef5bd5"
server_key_iv: "4cf6d29b3ca75cd73bad96cf65525a1d"
server_key_version: "kuduclusterkey@0"
I20260620 19:06:46.887413 12840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:06:46.924569 12840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:06:46.929535 12840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:06:46.930529 12840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:06:46.939816 12840 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:46053
I20260620 19:06:46.939900 12907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:46053 every 8 connection(s)
I20260620 19:06:46.941075 12840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:06:46.944001 12908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:06:46.949661 12908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599: Bootstrap starting.
I20260620 19:06:46.950496 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 12840
I20260620 19:06:46.950690 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:06:46.951126 1642 external_mini_cluster.cc:1428] Setting key 61c7519752190d638fa8bd8096c571ff
I20260620 19:06:46.953510 12908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599: Neither blocks nor log segments found. Creating new log.
I20260620 19:06:46.954363 12908 log.cc:824] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599: Log is configured to *not* fsync() on all Append() calls
I20260620 19:06:46.956535 12908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599: No bootstrap required, opened a new log
I20260620 19:06:46.960954 12908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e8a2b2ad3e49108b2250218cdee599" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46053 } }
I20260620 19:06:46.961279 12908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:06:46.961341 12908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98e8a2b2ad3e49108b2250218cdee599, State: Initialized, Role: FOLLOWER
I20260620 19:06:46.961869 12908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e8a2b2ad3e49108b2250218cdee599" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46053 } }
I20260620 19:06:46.962033 12908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:06:46.962107 12908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:06:46.962199 12908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 19:06:46 dist-test-slave-6wrf krb5kdc[12396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:06:46.963589 12908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e8a2b2ad3e49108b2250218cdee599" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46053 } }
I20260620 19:06:46.963997 12908 leader_election.cc:302] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98e8a2b2ad3e49108b2250218cdee599; no voters:
I20260620 19:06:46.964397 12908 leader_election.cc:288] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:06:46.965094 12913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:06:46.965858 12908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:06:46.965867 12913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [term 1 LEADER]: Becoming Leader. State: Replica: 98e8a2b2ad3e49108b2250218cdee599, State: Running, Role: LEADER
I20260620 19:06:46.966200 12913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e8a2b2ad3e49108b2250218cdee599" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46053 } }
I20260620 19:06:46.967810 12914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98e8a2b2ad3e49108b2250218cdee599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e8a2b2ad3e49108b2250218cdee599" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46053 } } }
I20260620 19:06:46.967935 12914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [sys.catalog]: This master's current role is: LEADER
I20260620 19:06:46.968227 12915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98e8a2b2ad3e49108b2250218cdee599. Latest consensus state: current_term: 1 leader_uuid: "98e8a2b2ad3e49108b2250218cdee599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e8a2b2ad3e49108b2250218cdee599" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46053 } } }
I20260620 19:06:46.968307 12915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599 [sys.catalog]: This master's current role is: LEADER
I20260620 19:06:46.968538 12923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:06:46.968940 12908 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:06:46.970491 12911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:46.952938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.1.154.190:46053)
0620 19:06:46.953442 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 19:06:46.953463 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 19:06:46.953508 (+ 45us) server_negotiation.cc:407] Connection header received
0620 19:06:46.954420 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:06:46.954443 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:06:46.954862 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:06:46.955278 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:06:46.956152 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:46.957636 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:46.958967 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:46.959701 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:46.959869 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:46.963052 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:06:46.963058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:06:46.963075 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:06:46.963106 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:06:46.966609 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:06:46.967225 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:06:46.967231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:06:46.967239 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:06:46.967382 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:06:46.967708 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:06:46.967711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:06:46.967713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:06:46.968302 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:06:46.969114 (+ 812us) server_negotiation.cc:1035] Waiting for connection context
0620 19:06:46.969262 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 19:06:46.969531 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":158,"threads_started":1}
I20260620 19:06:46.977279 12923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:06:46.986058 12923 catalog_manager.cc:1347] Generated new cluster ID: 7ff042cc77d94a2aa4da21d38d7a3f44
I20260620 19:06:46.986136 12923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:06:47.009176 12923 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:06:47.010564 12923 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:06:47.024159 12923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599: Generated new TSK 0
I20260620 19:06:47.025075 12923 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:06:47.817211 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.154.148:46969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46053
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:06:47.980351 12953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:06:47.980911 12953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:06:47.981055 12953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:06:47.986766 12953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:06:47.986827 12953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:06:47.986910 12953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:06:47.994592 12953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.154.148:46969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:06:47.996094 12953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:06:47.997498 12953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:06:48.009829 12961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:06:48.009843 12963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:06:48.009974 12960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:06:48.010602 12953 server_base.cc:1034] running on GCE node
I20260620 19:06:48.011480 12953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:06:48.012720 12953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:06:48.014007 12953 hybrid_clock.cc:648] HybridClock initialized: now 1781982408013969 us; error 57 us; skew 500 ppm
Jun 20 19:06:48 dist-test-slave-6wrf krb5kdc[12396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:06:48.017633 12953 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:06:48.018754 12953 webserver.cc:460] Webserver started at http://127.1.154.129:36559/ using document root <none> and password file <none>
I20260620 19:06:48.019479 12953 fs_manager.cc:359] Metadata directory not provided
I20260620 19:06:48.019582 12953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:06:48.020295 12953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:06:48 dist-test-slave-6wrf krb5kdc[12396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
I20260620 19:06:48.036098 12953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "a4bcd9884b164344af0eee55a5d025f3"
format_stamp: "Formatted at 2026-06-20 19:06:48 on dist-test-slave-6wrf"
server_key: "10e47db7933ab633f25ed8f73e50c682"
server_key_iv: "0de0265770faed97932f167ff6ae195d"
server_key_version: "kuduclusterkey@0"
I20260620 19:06:48.036767 12953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "a4bcd9884b164344af0eee55a5d025f3"
format_stamp: "Formatted at 2026-06-20 19:06:48 on dist-test-slave-6wrf"
server_key: "10e47db7933ab633f25ed8f73e50c682"
server_key_iv: "0de0265770faed97932f167ff6ae195d"
server_key_version: "kuduclusterkey@0"
I20260620 19:06:48.043305 12953 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 19:06:48.067028 12970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:06:48.069775 12953 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 19:06:48.070106 12953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "a4bcd9884b164344af0eee55a5d025f3"
format_stamp: "Formatted at 2026-06-20 19:06:48 on dist-test-slave-6wrf"
server_key: "10e47db7933ab633f25ed8f73e50c682"
server_key_iv: "0de0265770faed97932f167ff6ae195d"
server_key_version: "kuduclusterkey@0"
I20260620 19:06:48.070555 12953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:06:48.091260 12953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:06:48.095950 12953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:06:48.096254 12953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:06:48.097040 12953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:06:48.098214 12953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:06:48.098320 12953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:06:48.098449 12953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:06:48.098546 12953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:06:48.112488 12953 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:33923
I20260620 19:06:48.113229 13084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:33923 every 8 connection(s)
I20260620 19:06:48.114009 12953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:06:48.117280 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 12953
I20260620 19:06:48.117449 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:06:48.117861 1642 external_mini_cluster.cc:1428] Setting key 3ace579db9109c19d874f2dd147aeca8
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
Jun 20 19:06:48 dist-test-slave-6wrf krb5kdc[12396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:06:48.149472 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:48.117110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:37765 (local address 127.1.154.190:46053)
0620 19:06:48.129821 (+ 12711us) server_negotiation.cc:206] Beginning negotiation
0620 19:06:48.129827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:06:48.131848 (+ 2021us) server_negotiation.cc:407] Connection header received
0620 19:06:48.131908 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:06:48.131911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:06:48.131977 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:06:48.132066 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:06:48.134401 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:48.135769 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:48.140755 (+ 4986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:48.141151 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:48.141242 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:48.144742 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:06:48.144747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:06:48.144750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:06:48.144780 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:06:48.146760 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:06:48.147427 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:06:48.147431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:06:48.147433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:06:48.147526 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:06:48.147976 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:06:48.147979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:06:48.147981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:06:48.148221 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:06:48.148308 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:06:48.149126 (+ 818us) server_negotiation.cc:299] Negotiation successful
0620 19:06:48.149272 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12594,"thread_start_us":94,"threads_started":1}
I20260620 19:06:48.151719 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:48.117738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:46053 (local address 127.1.154.129:37765)
0620 19:06:48.130195 (+ 12457us) negotiation.cc:107] Waiting for socket to connect
0620 19:06:48.130276 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 19:06:48.131473 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:06:48.132858 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:06:48.132868 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:06:48.133239 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:06:48.134045 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:48.134066 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:48.138390 (+ 4324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:48.138394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:48.140303 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:48.140331 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:48.141502 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:48.141506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:48.141826 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:48.142632 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:06:48.142669 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:06:48.144500 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:06:48.146917 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:06:48.146923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:06:48.146942 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:06:48.147303 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:06:48.147640 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:06:48.147644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:06:48.147647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:06:48.147859 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:06:48.148392 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:06:48.148401 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:06:48.148803 (+ 402us) client_negotiation.cc:769] Sending connection context
0620 19:06:48.150246 (+ 1443us) client_negotiation.cc:240] Negotiation successful
0620 19:06:48.150624 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12156,"thread_start_us":137,"threads_started":1}
I20260620 19:06:48.156389 13085 heartbeater.cc:347] Connected to a master server at 127.1.154.190:46053
I20260620 19:06:48.156736 13085 heartbeater.cc:464] Registering TS with master...
I20260620 19:06:48.157401 13085 heartbeater.cc:511] Master 127.1.154.190:46053 requested a full tablet report, sending...
I20260620 19:06:48.159217 12872 ts_manager.cc:194] Registered new tserver with Master: a4bcd9884b164344af0eee55a5d025f3 (127.1.154.129:33923)
I20260620 19:06:48.160821 12872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:37765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:06:48.177302 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.154.148:46969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46053
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:06:48 dist-test-slave-6wrf krb5kdc[12396](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 19:06:48.361415 13092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:06:48.361872 13092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:06:48.361958 13092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:06:48.366928 13092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:06:48.367080 13092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:06:48.367218 13092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:06:48.375480 13092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.154.148:46969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:06:48.376834 13092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:06:48.378299 13092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:06:48.389638 13101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:06:48.389786 13099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:06:48.389832 13098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:06:48.391042 13092 server_base.cc:1034] running on GCE node
I20260620 19:06:48.391532 13092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:06:48.392236 13092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:06:48.395627 13092 hybrid_clock.cc:648] HybridClock initialized: now 1781982408395499 us; error 178 us; skew 500 ppm
Jun 20 19:06:48 dist-test-slave-6wrf krb5kdc[12396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:06:48.399454 13092 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:06:48.400653 13092 webserver.cc:460] Webserver started at http://127.1.154.130:32983/ using document root <none> and password file <none>
I20260620 19:06:48.401445 13092 fs_manager.cc:359] Metadata directory not provided
I20260620 19:06:48.401626 13092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:06:48.401925 13092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:06:48 dist-test-slave-6wrf krb5kdc[12396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
I20260620 19:06:48.425235 13092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "95558ba08ab244fbbda19ad47365f001"
format_stamp: "Formatted at 2026-06-20 19:06:48 on dist-test-slave-6wrf"
server_key: "5e78ad2e4fad725afa866cdfae1b8ba8"
server_key_iv: "b28fef14d9403c780317167858151d3b"
server_key_version: "kuduclusterkey@0"
I20260620 19:06:48.428248 13092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "95558ba08ab244fbbda19ad47365f001"
format_stamp: "Formatted at 2026-06-20 19:06:48 on dist-test-slave-6wrf"
server_key: "5e78ad2e4fad725afa866cdfae1b8ba8"
server_key_iv: "b28fef14d9403c780317167858151d3b"
server_key_version: "kuduclusterkey@0"
I20260620 19:06:48.434036 13092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 19:06:48.458173 13108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:06:48.460073 13092 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260620 19:06:48.460315 13092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "95558ba08ab244fbbda19ad47365f001"
format_stamp: "Formatted at 2026-06-20 19:06:48 on dist-test-slave-6wrf"
server_key: "5e78ad2e4fad725afa866cdfae1b8ba8"
server_key_iv: "b28fef14d9403c780317167858151d3b"
server_key_version: "kuduclusterkey@0"
I20260620 19:06:48.460559 13092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:06:48.501721 13092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:06:48.508181 13092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:06:48.508451 13092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:06:48.509243 13092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:06:48.510440 13092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:06:48.510577 13092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:06:48.510684 13092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:06:48.510764 13092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:06:48.521620 13092 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:45059
I20260620 19:06:48.522832 13092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:06:48.528734 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 13092
I20260620 19:06:48.528901 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:06:48.529273 1642 external_mini_cluster.cc:1428] Setting key 7452870465875870d0ac46f58431a182
I20260620 19:06:48.534631 13221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:45059 every 8 connection(s)
Jun 20 19:06:48 dist-test-slave-6wrf krb5kdc[12396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:06:48.558386 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:48.542251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:47435 (local address 127.1.154.190:46053)
0620 19:06:48.542433 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 19:06:48.542440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:06:48.542892 (+ 452us) server_negotiation.cc:407] Connection header received
0620 19:06:48.544300 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:06:48.544304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:06:48.544363 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:06:48.544420 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:06:48.546307 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:48.547207 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:48.549267 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:48.549688 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:48.549777 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:48.554052 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:06:48.554057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:06:48.554059 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:06:48.554089 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:06:48.555913 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:06:48.556571 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:06:48.556575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:06:48.556577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:06:48.556658 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:06:48.557049 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:06:48.557052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:06:48.557054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:06:48.557270 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:06:48.557356 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:06:48.558079 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 19:06:48.558214 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
I20260620 19:06:48.560343 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:48.540265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:46053 (local address 127.1.154.130:47435)
0620 19:06:48.542689 (+ 2424us) negotiation.cc:107] Waiting for socket to connect
0620 19:06:48.542767 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 19:06:48.544015 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:06:48.544917 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:06:48.544928 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:06:48.545328 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:06:48.546120 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:48.546140 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:48.547382 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:48.547386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:48.549117 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:48.549128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:48.550015 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:48.550020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:48.550239 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:48.551037 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:06:48.551060 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:06:48.553826 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:06:48.556074 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:06:48.556080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:06:48.556097 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:06:48.556443 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:06:48.556760 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:06:48.556763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:06:48.556765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:06:48.556940 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:06:48.557424 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:06:48.557430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:06:48.557795 (+ 365us) client_negotiation.cc:769] Sending connection context
0620 19:06:48.558916 (+ 1121us) client_negotiation.cc:240] Negotiation successful
0620 19:06:48.559270 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2147,"thread_start_us":472,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
I20260620 19:06:48.562043 13222 heartbeater.cc:347] Connected to a master server at 127.1.154.190:46053
I20260620 19:06:48.562292 13222 heartbeater.cc:464] Registering TS with master...
I20260620 19:06:48.562911 13222 heartbeater.cc:511] Master 127.1.154.190:46053 requested a full tablet report, sending...
I20260620 19:06:48.564112 12872 ts_manager.cc:194] Registered new tserver with Master: 95558ba08ab244fbbda19ad47365f001 (127.1.154.130:45059)
I20260620 19:06:48.564908 12872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:47435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:06:48.586802 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.154.148:46969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46053
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:06:48.759286 13227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:06:48.759689 13227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:06:48.759759 13227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:06:48.766448 13227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:06:48.766700 13227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:06:48.766827 13227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:06:48.775473 13227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.154.148:46969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:06:48.777803 13227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:06:48.779325 13227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 20 19:06:48 dist-test-slave-6wrf krb5kdc[12396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
W20260620 19:06:48.794170 13232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:06:48.794642 13227 server_base.cc:1034] running on GCE node
W20260620 19:06:48.795109 13235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:06:48.798286 13233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:06:48.798951 13227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:06:48.799816 13227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:06:48.831429 13227 hybrid_clock.cc:648] HybridClock initialized: now 1781982408831356 us; error 86 us; skew 500 ppm
Jun 20 19:06:48 dist-test-slave-6wrf krb5kdc[12396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:06:48.835544 13227 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:06:48.836766 13227 webserver.cc:460] Webserver started at http://127.1.154.131:41297/ using document root <none> and password file <none>
I20260620 19:06:48.837491 13227 fs_manager.cc:359] Metadata directory not provided
I20260620 19:06:48.837563 13227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:06:48.837819 13227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:06:48 dist-test-slave-6wrf krb5kdc[12396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
I20260620 19:06:48.852612 13227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "6b2fb8d8101444ea9df2034913f86cc8"
format_stamp: "Formatted at 2026-06-20 19:06:48 on dist-test-slave-6wrf"
server_key: "a0a9ad98541a6b1d6ca2aa4926709104"
server_key_iv: "bb10d93aa020fda660b056b47c28f568"
server_key_version: "kuduclusterkey@0"
I20260620 19:06:48.853390 13227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "6b2fb8d8101444ea9df2034913f86cc8"
format_stamp: "Formatted at 2026-06-20 19:06:48 on dist-test-slave-6wrf"
server_key: "a0a9ad98541a6b1d6ca2aa4926709104"
server_key_iv: "bb10d93aa020fda660b056b47c28f568"
server_key_version: "kuduclusterkey@0"
I20260620 19:06:48.868721 13227 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.003s sys 0.001s
I20260620 19:06:48.900349 13242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:06:48.902294 13227 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260620 19:06:48.902427 13227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "6b2fb8d8101444ea9df2034913f86cc8"
format_stamp: "Formatted at 2026-06-20 19:06:48 on dist-test-slave-6wrf"
server_key: "a0a9ad98541a6b1d6ca2aa4926709104"
server_key_iv: "bb10d93aa020fda660b056b47c28f568"
server_key_version: "kuduclusterkey@0"
I20260620 19:06:48.902573 13227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:06:48.937037 13227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:06:48.940850 13227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:06:48.941116 13227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:06:48.941928 13227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:06:48.943186 13227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:06:48.943295 13227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:06:48.943390 13227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:06:48.943451 13227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:06:48.956591 13227 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:40993
I20260620 19:06:48.957851 13227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:06:48.958431 13355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:40993 every 8 connection(s)
I20260620 19:06:48.962208 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 13227
I20260620 19:06:48.962327 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:06:48.962613 1642 external_mini_cluster.cc:1428] Setting key 8a8387b27e304137468880630c5abb2e
Jun 20 19:06:48 dist-test-slave-6wrf krb5kdc[12396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:06:48.980859 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:48.961793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:41645 (local address 127.1.154.190:46053)
0620 19:06:48.961927 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 19:06:48.961931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:06:48.965824 (+ 3893us) server_negotiation.cc:407] Connection header received
0620 19:06:48.967022 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:06:48.967025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:06:48.967087 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:06:48.967141 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:06:48.968952 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:48.969802 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:48.971437 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:48.971770 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:48.971837 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:48.976590 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:06:48.976594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:06:48.976597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:06:48.976623 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:06:48.978414 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:06:48.979050 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:06:48.979053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:06:48.979067 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:06:48.979149 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:06:48.979563 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:06:48.979566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:06:48.979568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:06:48.979795 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:06:48.979863 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:06:48.980558 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 19:06:48.980681 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:06:48.983243 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:48.961339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:46053 (local address 127.1.154.131:41645)
0620 19:06:48.965681 (+ 4342us) negotiation.cc:107] Waiting for socket to connect
0620 19:06:48.965737 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 19:06:48.966791 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:06:48.967557 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:06:48.967565 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:06:48.967913 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:06:48.968565 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:48.968582 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:48.969948 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:48.969951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:48.971305 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:48.971315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:48.972023 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:48.972027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:48.972368 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:48.973124 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:06:48.973144 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:06:48.976403 (+ 3259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:06:48.978563 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:06:48.978570 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:06:48.978588 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:06:48.978936 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:06:48.979254 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:06:48.979256 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:06:48.979258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:06:48.979442 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:06:48.979941 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:06:48.979948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:06:48.980286 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 19:06:48.981621 (+ 1335us) client_negotiation.cc:240] Negotiation successful
0620 19:06:48.982191 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4099,"thread_start_us":150,"threads_started":1}
I20260620 19:06:48.984709 13356 heartbeater.cc:347] Connected to a master server at 127.1.154.190:46053
I20260620 19:06:48.984942 13356 heartbeater.cc:464] Registering TS with master...
I20260620 19:06:48.985487 13356 heartbeater.cc:511] Master 127.1.154.190:46053 requested a full tablet report, sending...
I20260620 19:06:48.988379 12872 ts_manager.cc:194] Registered new tserver with Master: 6b2fb8d8101444ea9df2034913f86cc8 (127.1.154.131:40993)
I20260620 19:06:48.989125 12872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:41645
I20260620 19:06:48.999430 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:06:49.012208 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.001920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59988 (local address 127.1.154.190:46053)
0620 19:06:49.002036 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:06:49.002040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:06:49.002534 (+ 494us) server_negotiation.cc:407] Connection header received
0620 19:06:49.002708 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:06:49.002711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:06:49.002755 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:06:49.002827 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:06:49.003770 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.004588 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.005961 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.006305 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.006376 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.008551 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:06:49.008555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:06:49.008558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:06:49.008586 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:06:49.010356 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:06:49.010884 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:06:49.010887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:06:49.010889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:06:49.010970 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:06:49.011264 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:06:49.011267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:06:49.011269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:06:49.011473 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:06:49.011570 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:06:49.011905 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 19:06:49.012026 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:06:49.016224 12872 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:06:49.020741 12872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:06:49.043277 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.032069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:33923 (local address 127.0.0.1:55404)
0620 19:06:49.036927 (+ 4858us) negotiation.cc:107] Waiting for socket to connect
0620 19:06:49.036998 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 19:06:49.037130 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:06:49.038331 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:06:49.038335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:06:49.038365 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:06:49.038602 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:49.038609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.040214 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.040218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:49.041665 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:49.041675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.042734 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.042738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:49.042858 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.042897 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 19:06:49.043002 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 19:06:49.043117 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4671,"spinlock_wait_cycles":2944,"thread_start_us":98,"threads_started":1}
I20260620 19:06:49.044279 13370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.032230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.1.154.129:33923)
0620 19:06:49.037653 (+ 5423us) server_negotiation.cc:206] Beginning negotiation
0620 19:06:49.037660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:06:49.037687 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:06:49.037764 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:06:49.037770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:06:49.038001 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:06:49.038172 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:06:49.038752 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.039903 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.041815 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.042633 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.042741 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.042888 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 19:06:49.044022 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0620 19:06:49.044137 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5288,"thread_start_us":68,"threads_started":1}
I20260620 19:06:49.047885 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.034169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:40993 (local address 127.0.0.1:43340)
0620 19:06:49.037551 (+ 3382us) negotiation.cc:107] Waiting for socket to connect
0620 19:06:49.037562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:06:49.037673 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:06:49.038321 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:06:49.038327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:06:49.038347 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:06:49.039206 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:49.039215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.040730 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.040734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:49.045221 (+ 4487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:49.045231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.047269 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.047272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:49.047368 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.047390 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:06:49.047459 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 19:06:49.047528 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3325,"mutex_wait_us":3242,"spinlock_wait_cycles":1280,"thread_start_us":98,"threads_started":1}
I20260620 19:06:49.048799 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.034604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.1.154.131:40993)
0620 19:06:49.037414 (+ 2810us) server_negotiation.cc:206] Beginning negotiation
0620 19:06:49.037425 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:06:49.037760 (+ 335us) server_negotiation.cc:407] Connection header received
0620 19:06:49.037835 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:06:49.037840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:06:49.038025 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:06:49.038163 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:06:49.039368 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.040599 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.045380 (+ 4781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.046964 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.047073 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.047206 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 19:06:49.048351 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0620 19:06:49.048650 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2704,"thread_start_us":98,"threads_started":1}
I20260620 19:06:49.048230 13007 tablet_service.cc:1467] Processing CreateTablet for tablet 48e6ac412eb342808cfad27377f1ebd9 (DEFAULT_TABLE table=test-table [id=856eee7b6b924df9ba7f916ed7cdb2d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:06:49.049208 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.035835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.1.154.130:45059)
0620 19:06:49.043116 (+ 7281us) server_negotiation.cc:206] Beginning negotiation
0620 19:06:49.043133 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:06:49.043158 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:06:49.043236 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:06:49.043241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:06:49.043458 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:06:49.043622 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:06:49.044013 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.045939 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.047396 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.048186 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.048293 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.048411 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 19:06:49.048957 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 19:06:49.049067 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7152,"thread_start_us":114,"threads_started":1}
I20260620 19:06:49.048858 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.035008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:45059 (local address 127.0.0.1:58170)
0620 19:06:49.042436 (+ 7428us) negotiation.cc:107] Waiting for socket to connect
0620 19:06:49.042450 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:06:49.042546 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:06:49.043769 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:06:49.043772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:06:49.043790 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:06:49.043870 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:49.043879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.046087 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.046090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:49.047231 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:49.047242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.048469 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.048473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:49.048565 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.048587 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:06:49.048654 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:06:49.048719 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7358,"thread_start_us":96,"threads_started":1}
I20260620 19:06:49.049664 13007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e6ac412eb342808cfad27377f1ebd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:06:49.052246 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 48e6ac412eb342808cfad27377f1ebd9 (DEFAULT_TABLE table=test-table [id=856eee7b6b924df9ba7f916ed7cdb2d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:06:49.052250 13156 tablet_service.cc:1467] Processing CreateTablet for tablet 48e6ac412eb342808cfad27377f1ebd9 (DEFAULT_TABLE table=test-table [id=856eee7b6b924df9ba7f916ed7cdb2d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:06:49.053331 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e6ac412eb342808cfad27377f1ebd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:06:49.053398 13156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e6ac412eb342808cfad27377f1ebd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:06:49.063118 13377 tablet_bootstrap.cc:492] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8: Bootstrap starting.
I20260620 19:06:49.063225 13375 tablet_bootstrap.cc:492] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3: Bootstrap starting.
I20260620 19:06:49.063546 13376 tablet_bootstrap.cc:492] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001: Bootstrap starting.
I20260620 19:06:49.065763 13375 tablet_bootstrap.cc:654] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3: Neither blocks nor log segments found. Creating new log.
I20260620 19:06:49.065763 13376 tablet_bootstrap.cc:654] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001: Neither blocks nor log segments found. Creating new log.
I20260620 19:06:49.066726 13375 log.cc:824] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:06:49.069126 13375 tablet_bootstrap.cc:492] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3: No bootstrap required, opened a new log
I20260620 19:06:49.069315 13375 ts_tablet_manager.cc:1397] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20260620 19:06:49.069520 13377 tablet_bootstrap.cc:654] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8: Neither blocks nor log segments found. Creating new log.
I20260620 19:06:49.070394 13377 log.cc:824] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:06:49.071652 13376 log.cc:824] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001: Log is configured to *not* fsync() on all Append() calls
I20260620 19:06:49.074191 13375 raft_consensus.cc:348] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcd9884b164344af0eee55a5d025f3" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 33923 } } peers { permanent_uuid: "95558ba08ab244fbbda19ad47365f001" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45059 } } peers { permanent_uuid: "6b2fb8d8101444ea9df2034913f86cc8" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 40993 } }
I20260620 19:06:49.074626 13375 raft_consensus.cc:374] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:06:49.074731 13375 raft_consensus.cc:729] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bcd9884b164344af0eee55a5d025f3, State: Initialized, Role: FOLLOWER
I20260620 19:06:49.075304 13375 consensus_queue.cc:260] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcd9884b164344af0eee55a5d025f3" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 33923 } } peers { permanent_uuid: "95558ba08ab244fbbda19ad47365f001" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45059 } } peers { permanent_uuid: "6b2fb8d8101444ea9df2034913f86cc8" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 40993 } }
I20260620 19:06:49.075603 13376 tablet_bootstrap.cc:492] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001: No bootstrap required, opened a new log
I20260620 19:06:49.075851 13376 ts_tablet_manager.cc:1397] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20260620 19:06:49.076571 13375 ts_tablet_manager.cc:1428] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260620 19:06:49.076954 13085 heartbeater.cc:503] Master 127.1.154.190:46053 was elected leader, sending a full tablet report...
I20260620 19:06:49.079754 13376 raft_consensus.cc:348] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcd9884b164344af0eee55a5d025f3" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 33923 } } peers { permanent_uuid: "95558ba08ab244fbbda19ad47365f001" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45059 } } peers { permanent_uuid: "6b2fb8d8101444ea9df2034913f86cc8" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 40993 } }
I20260620 19:06:49.080137 13376 raft_consensus.cc:374] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:06:49.080263 13376 raft_consensus.cc:729] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95558ba08ab244fbbda19ad47365f001, State: Initialized, Role: FOLLOWER
I20260620 19:06:49.080814 13376 consensus_queue.cc:260] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcd9884b164344af0eee55a5d025f3" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 33923 } } peers { permanent_uuid: "95558ba08ab244fbbda19ad47365f001" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45059 } } peers { permanent_uuid: "6b2fb8d8101444ea9df2034913f86cc8" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 40993 } }
I20260620 19:06:49.082784 13377 tablet_bootstrap.cc:492] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8: No bootstrap required, opened a new log
I20260620 19:06:49.083034 13377 ts_tablet_manager.cc:1397] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260620 19:06:49.083516 13376 ts_tablet_manager.cc:1428] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260620 19:06:49.083923 13222 heartbeater.cc:503] Master 127.1.154.190:46053 was elected leader, sending a full tablet report...
I20260620 19:06:49.087803 13377 raft_consensus.cc:348] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcd9884b164344af0eee55a5d025f3" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 33923 } } peers { permanent_uuid: "95558ba08ab244fbbda19ad47365f001" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45059 } } peers { permanent_uuid: "6b2fb8d8101444ea9df2034913f86cc8" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 40993 } }
I20260620 19:06:49.088166 13377 raft_consensus.cc:374] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:06:49.088263 13377 raft_consensus.cc:729] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2fb8d8101444ea9df2034913f86cc8, State: Initialized, Role: FOLLOWER
I20260620 19:06:49.088843 13377 consensus_queue.cc:260] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcd9884b164344af0eee55a5d025f3" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 33923 } } peers { permanent_uuid: "95558ba08ab244fbbda19ad47365f001" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45059 } } peers { permanent_uuid: "6b2fb8d8101444ea9df2034913f86cc8" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 40993 } }
I20260620 19:06:49.089540 13356 heartbeater.cc:503] Master 127.1.154.190:46053 was elected leader, sending a full tablet report...
I20260620 19:06:49.089964 13377 ts_tablet_manager.cc:1428] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:06:49.110220 13382 raft_consensus.cc:482] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:06:49.110462 13382 raft_consensus.cc:504] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcd9884b164344af0eee55a5d025f3" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 33923 } } peers { permanent_uuid: "95558ba08ab244fbbda19ad47365f001" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45059 } } peers { permanent_uuid: "6b2fb8d8101444ea9df2034913f86cc8" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 40993 } }
I20260620 19:06:49.111887 13382 leader_election.cc:288] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4bcd9884b164344af0eee55a5d025f3 (127.1.154.129:33923), 6b2fb8d8101444ea9df2034913f86cc8 (127.1.154.131:40993)
I20260620 19:06:49.113059 13381 raft_consensus.cc:482] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:06:49.113209 13381 raft_consensus.cc:504] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcd9884b164344af0eee55a5d025f3" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 33923 } } peers { permanent_uuid: "95558ba08ab244fbbda19ad47365f001" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45059 } } peers { permanent_uuid: "6b2fb8d8101444ea9df2034913f86cc8" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 40993 } }
I20260620 19:06:49.114251 13381 leader_election.cc:288] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95558ba08ab244fbbda19ad47365f001 (127.1.154.130:45059), 6b2fb8d8101444ea9df2034913f86cc8 (127.1.154.131:40993)
W20260620 19:06:49.120185 13086 tablet.cc:2369] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:06:49.125653 13370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.112924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:60341 (local address 127.1.154.129:33923)
0620 19:06:49.117713 (+ 4789us) server_negotiation.cc:206] Beginning negotiation
0620 19:06:49.117719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:06:49.117735 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:06:49.117787 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:06:49.117790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:06:49.117844 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:06:49.117939 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:06:49.122094 (+ 4155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.122810 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.124250 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.124796 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.124891 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.124953 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:06:49.125430 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 19:06:49.125506 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4700}
I20260620 19:06:49.125669 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.112100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:33923 (local address 127.1.154.130:60341)
0620 19:06:49.113719 (+ 1619us) negotiation.cc:107] Waiting for socket to connect
0620 19:06:49.113734 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:06:49.113836 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:06:49.118029 (+ 4193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:06:49.118032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:06:49.118052 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:06:49.118166 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:49.118171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.122971 (+ 4800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.122974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:49.124111 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:49.124122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.125173 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.125176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:49.125288 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.125313 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 19:06:49.125389 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:06:49.125464 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1551,"thread_start_us":109,"threads_started":1}
I20260620 19:06:49.126497 13023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e6ac412eb342808cfad27377f1ebd9" candidate_uuid: "95558ba08ab244fbbda19ad47365f001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bcd9884b164344af0eee55a5d025f3" is_pre_election: true
I20260620 19:06:49.126748 13023 raft_consensus.cc:2454] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95558ba08ab244fbbda19ad47365f001 in term 0.
I20260620 19:06:49.127390 13111 leader_election.cc:302] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95558ba08ab244fbbda19ad47365f001, a4bcd9884b164344af0eee55a5d025f3; no voters:
I20260620 19:06:49.127722 13382 raft_consensus.cc:2781] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:06:49.127799 13382 raft_consensus.cc:482] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:06:49.127842 13382 raft_consensus.cc:3037] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:06:49.128048 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.113082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:52563 (local address 127.1.154.131:40993)
0620 19:06:49.113206 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:06:49.113210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:06:49.114123 (+ 913us) server_negotiation.cc:407] Connection header received
0620 19:06:49.114278 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:06:49.114281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:06:49.114325 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:06:49.114368 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:06:49.115037 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.115787 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.126675 (+ 10888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.127375 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.127490 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.127539 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:06:49.127844 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 19:06:49.127925 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:06:49.128423 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.112487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:40993 (local address 127.1.154.130:52563)
0620 19:06:49.114053 (+ 1566us) negotiation.cc:107] Waiting for socket to connect
0620 19:06:49.114064 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:06:49.114176 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:06:49.114497 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:06:49.114500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:06:49.114514 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:06:49.114633 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:49.114639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.116281 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.116285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:49.117890 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:49.117901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.127604 (+ 9703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.127608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:49.127711 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.127738 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:06:49.128220 (+ 482us) client_negotiation.cc:240] Negotiation successful
0620 19:06:49.128294 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1517,"thread_start_us":82,"threads_started":1}
I20260620 19:06:49.129253 13382 raft_consensus.cc:504] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcd9884b164344af0eee55a5d025f3" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 33923 } } peers { permanent_uuid: "95558ba08ab244fbbda19ad47365f001" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45059 } } peers { permanent_uuid: "6b2fb8d8101444ea9df2034913f86cc8" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 40993 } }
I20260620 19:06:49.129670 13382 leader_election.cc:288] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [CANDIDATE]: Term 1 election: Requested vote from peers a4bcd9884b164344af0eee55a5d025f3 (127.1.154.129:33923), 6b2fb8d8101444ea9df2034913f86cc8 (127.1.154.131:40993)
I20260620 19:06:49.130019 13023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e6ac412eb342808cfad27377f1ebd9" candidate_uuid: "95558ba08ab244fbbda19ad47365f001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bcd9884b164344af0eee55a5d025f3"
I20260620 19:06:49.130016 13295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e6ac412eb342808cfad27377f1ebd9" candidate_uuid: "95558ba08ab244fbbda19ad47365f001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2fb8d8101444ea9df2034913f86cc8" is_pre_election: true
I20260620 19:06:49.130134 13023 raft_consensus.cc:3037] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:06:49.130297 13295 raft_consensus.cc:2454] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95558ba08ab244fbbda19ad47365f001 in term 0.
I20260620 19:06:49.131443 13023 raft_consensus.cc:2454] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95558ba08ab244fbbda19ad47365f001 in term 1.
I20260620 19:06:49.131578 13295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e6ac412eb342808cfad27377f1ebd9" candidate_uuid: "95558ba08ab244fbbda19ad47365f001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2fb8d8101444ea9df2034913f86cc8"
I20260620 19:06:49.131716 13295 raft_consensus.cc:3037] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:06:49.131827 13111 leader_election.cc:302] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95558ba08ab244fbbda19ad47365f001, a4bcd9884b164344af0eee55a5d025f3; no voters:
I20260620 19:06:49.133076 13295 raft_consensus.cc:2454] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95558ba08ab244fbbda19ad47365f001 in term 1.
Jun 20 19:06:49 dist-test-slave-6wrf krb5kdc[12396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
I20260620 19:06:49.134020 13382 raft_consensus.cc:2781] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:06:49.134261 13382 raft_consensus.cc:686] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [term 1 LEADER]: Becoming Leader. State: Replica: 95558ba08ab244fbbda19ad47365f001, State: Running, Role: LEADER
I20260620 19:06:49.134614 13382 consensus_queue.cc:237] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcd9884b164344af0eee55a5d025f3" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 33923 } } peers { permanent_uuid: "95558ba08ab244fbbda19ad47365f001" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45059 } } peers { permanent_uuid: "6b2fb8d8101444ea9df2034913f86cc8" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 40993 } }
I20260620 19:06:49.138409 12872 catalog_manager.cc:5495] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95558ba08ab244fbbda19ad47365f001 (127.1.154.130). New cstate: current_term: 1 leader_uuid: "95558ba08ab244fbbda19ad47365f001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcd9884b164344af0eee55a5d025f3" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95558ba08ab244fbbda19ad47365f001" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b2fb8d8101444ea9df2034913f86cc8" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 40993 } health_report { overall_health: UNKNOWN } } }
I20260620 19:06:49.139775 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.115620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:40993 (local address 127.1.154.129:39163)
0620 19:06:49.132265 (+ 16645us) negotiation.cc:107] Waiting for socket to connect
0620 19:06:49.132279 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:06:49.132375 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:06:49.132745 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:06:49.132751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:06:49.132773 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:06:49.132889 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:49.132895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.133959 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.133963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:49.135123 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:49.135133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.139308 (+ 4175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.139312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:49.139416 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.139444 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 19:06:49.139522 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:06:49.139604 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16574,"thread_start_us":117,"threads_started":1}
I20260620 19:06:49.140234 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.116028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:39163 (local address 127.1.154.131:40993)
0620 19:06:49.126763 (+ 10735us) server_negotiation.cc:206] Beginning negotiation
0620 19:06:49.126775 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:06:49.132453 (+ 5678us) server_negotiation.cc:407] Connection header received
0620 19:06:49.132498 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:06:49.132500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:06:49.132538 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:06:49.132610 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:06:49.133021 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.133825 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.135256 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.135843 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.135951 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.135994 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 19:06:49.139993 (+ 3999us) server_negotiation.cc:299] Negotiation successful
0620 19:06:49.140075 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10651,"thread_start_us":105,"threads_started":1}
I20260620 19:06:49.140560 13295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e6ac412eb342808cfad27377f1ebd9" candidate_uuid: "a4bcd9884b164344af0eee55a5d025f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2fb8d8101444ea9df2034913f86cc8" is_pre_election: true
I20260620 19:06:49.140741 13295 raft_consensus.cc:2379] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4bcd9884b164344af0eee55a5d025f3 in current term 1: Already voted for candidate 95558ba08ab244fbbda19ad47365f001 in this term.
I20260620 19:06:49.151000 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.114951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:39145 (local address 127.1.154.130:45059)
0620 19:06:49.115052 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 19:06:49.115056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:06:49.127419 (+ 12363us) server_negotiation.cc:407] Connection header received
0620 19:06:49.127542 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:06:49.127547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:06:49.127604 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:06:49.127673 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:06:49.133949 (+ 6276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.146424 (+ 12475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.148406 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.149015 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.149139 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.149190 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:06:49.150753 (+ 1563us) server_negotiation.cc:299] Negotiation successful
0620 19:06:49.150836 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 19:06:49.151340 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.114939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:45059 (local address 127.1.154.129:39145)
0620 19:06:49.127338 (+ 12399us) negotiation.cc:107] Waiting for socket to connect
0620 19:06:49.127352 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:06:49.127456 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:06:49.133690 (+ 6234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:06:49.133693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:06:49.133711 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:06:49.133825 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:49.133830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.146597 (+ 12767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.146601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:49.147981 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:06:49.147994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.149258 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.149261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:06:49.149358 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.149382 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 19:06:49.149454 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:06:49.149534 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12331,"thread_start_us":166,"threads_started":1}
I20260620 19:06:49.153226 13157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e6ac412eb342808cfad27377f1ebd9" candidate_uuid: "a4bcd9884b164344af0eee55a5d025f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95558ba08ab244fbbda19ad47365f001" is_pre_election: true
I20260620 19:06:49.154110 12973 leader_election.cc:302] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4bcd9884b164344af0eee55a5d025f3; no voters: 6b2fb8d8101444ea9df2034913f86cc8, 95558ba08ab244fbbda19ad47365f001
I20260620 19:06:49.154686 13381 raft_consensus.cc:2726] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:06:49.183807 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.177968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.1.154.130:45059)
0620 19:06:49.178106 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:06:49.178110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:06:49.178225 (+ 115us) server_negotiation.cc:407] Connection header received
0620 19:06:49.178398 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:06:49.178401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:06:49.178468 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:06:49.178566 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:06:49.178931 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.179695 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.180866 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:06:49.181237 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:06:49.181337 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:06:49.182107 (+ 770us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:06:49.182975 (+ 868us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:06:49.183108 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 19:06:49.183605 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 19:06:49.183664 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:06:49.191633 13295 raft_consensus.cc:1261] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8 [term 1 FOLLOWER]: Refusing update from remote peer 95558ba08ab244fbbda19ad47365f001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:06:49.192476 13023 raft_consensus.cc:1261] T 48e6ac412eb342808cfad27377f1ebd9 P a4bcd9884b164344af0eee55a5d025f3 [term 1 FOLLOWER]: Refusing update from remote peer 95558ba08ab244fbbda19ad47365f001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:06:49.193149 13382 consensus_queue.cc:1035] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bcd9884b164344af0eee55a5d025f3" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:06:49.193325 13390 consensus_queue.cc:1035] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2fb8d8101444ea9df2034913f86cc8" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 40993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:06:49.209256 13357 tablet.cc:2369] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:06:49.213451 13393 mvcc.cc:205] Tried to move back new op lower bound from 7298999948038496256 to 7298999947818561536. Current Snapshot: MvccSnapshot[applied={T|T < 7298999948038496256}]
I20260620 19:06:49.219717 13397 mvcc.cc:205] Tried to move back new op lower bound from 7298999948038496256 to 7298999947818561536. Current Snapshot: MvccSnapshot[applied={T|T < 7298999948038496256}]
I20260620 19:06:49.221287 13395 mvcc.cc:205] Tried to move back new op lower bound from 7298999948038496256 to 7298999947818561536. Current Snapshot: MvccSnapshot[applied={T|T < 7298999948038496256}]
I20260620 19:06:49.231189 12872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:06:49.231388 12872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:06:49.234431 12872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98e8a2b2ad3e49108b2250218cdee599: Sending DeleteTablet for 3 replicas of tablet 48e6ac412eb342808cfad27377f1ebd9
I20260620 19:06:49.235244 13007 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e6ac412eb342808cfad27377f1ebd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:06:49 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:55404
I20260620 19:06:49.236304 13156 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e6ac412eb342808cfad27377f1ebd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:06:49 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:58170
I20260620 19:06:49.237226 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e6ac412eb342808cfad27377f1ebd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:06:49 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:43340
I20260620 19:06:49.238620 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 12953
I20260620 19:06:49.241158 13404 tablet_replica.cc:330] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001: stopping tablet replica
I20260620 19:06:49.241518 13404 raft_consensus.cc:2229] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:06:49.241914 13404 raft_consensus.cc:2258] T 48e6ac412eb342808cfad27377f1ebd9 P 95558ba08ab244fbbda19ad47365f001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:06:49.242300 13405 tablet_replica.cc:330] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8: stopping tablet replica
I20260620 19:06:49.242596 13405 raft_consensus.cc:2229] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:06:49.242887 13405 raft_consensus.cc:2258] T 48e6ac412eb342808cfad27377f1ebd9 P 6b2fb8d8101444ea9df2034913f86cc8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 19:06:49.257776 12858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:33923 (error 108)
I20260620 19:06:49.258953 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 13092
I20260620 19:06:49.271055 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.270687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:33923 (local address 127.0.0.1:55410)
0620 19:06:49.270884 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 19:06:49.270966 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:33923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 19:06:49.271319 12858 catalog_manager.cc:4584] TS a4bcd9884b164344af0eee55a5d025f3 (127.1.154.129:33923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48e6ac412eb342808cfad27377f1ebd9: Network error: Client connection negotiation failed: client connection to 127.1.154.129:33923: connect: Connection refused (error 111)
I20260620 19:06:49.273763 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 13227
I20260620 19:06:49.274967 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:06:49.274667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:45059 (local address 127.0.0.1:58174)
0620 19:06:49.274831 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 19:06:49.274885 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.130:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 19:06:49.280339 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 12840
I20260620 19:06:49.287108 1642 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 19:06:49.623855 1642 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 19:06:49.623994 1642 mini_ranger.cc:66] Stopping Ranger...
I20260620 19:06:49.970935 1642 mini_ranger.cc:68] Stopped Ranger
2026-06-20 19:06:49.971 UTC [12426] LOG: received smart shutdown request
2026-06-20 19:06:49.975 UTC [12426] LOG: background worker "logical replication launcher" (PID 12435) exited with exit code 1
2026-06-20 19:06:49.975 UTC [12430] LOG: shutting down
2026-06-20 19:06:50.179 UTC [12426] LOG: database system is shut down
2026-06-20T19:06:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52977 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:06:50 dist-test-slave-6wrf krb5kdc[13415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:06:50 dist-test-slave-6wrf krb5kdc[13415](info): set up 2 sockets
Jun 20 19:06:50 dist-test-slave-6wrf krb5kdc[13415](info): commencing operation
krb5kdc: starting...
W20260620 19:06:51.389814 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:06:51 dist-test-slave-6wrf krb5kdc[13415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:06:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.154.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.154.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/rangeradmin_127.1.154.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/rangeradmin_127.1.154.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.154.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.154.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/rangerlookup_127.1.154.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/rangerlookup_127.1.154.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.154.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.154.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/HTTP_127.1.154.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/HTTP_127.1.154.148@KRBTEST.COM.keytab.
I20260620 19:06:51.585048 1642 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8BnTdh/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/postgres -l logfile start
2026-06-20 19:06:54.500 UTC [13445] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 19:06:54.500 UTC [13445] LOG: listening on IPv4 address "127.1.154.148", port 35583
2026-06-20 19:06:54.504 UTC [13445] LOG: listening on Unix socket "/tmp/.s.PGSQL.35583"
2026-06-20 19:06:54.526 UTC [13448] LOG: database system was shut down at 2026-06-20 19:06:52 UTC
2026-06-20 19:06:54.531 UTC [13445] LOG: database system is ready to accept connections
I20260620 19:06:55.518333 1642 mini_postgres.cc:96] Postgres bound to 35583
2026-06-20 19:06:55.524 UTC [13456] FATAL: database "slave" does not exist
127.1.154.148:35583 - accepting connections
I20260620 19:06:55.524956 1642 mini_ranger.cc:161] Starting Ranger...
I20260620 19:06:55.542089 1642 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 19:06:56.159937 1642 mini_ranger.cc:87] Created ranger Postgres database
I20260620 19:06:56.160076 1642 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-admin
2026-06-20 19:06:56,453 [I] DB FLAVOR :POSTGRES
2026-06-20 19:06:56,454 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:06:56,454 [I] Checking connection..
2026-06-20 19:06:56,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 19:06:56,726 [I] Checking connection passed.
2026-06-20 19:06:56,727 [I] --------- Verifying version history table ---------
2026-06-20 19:06:56,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:06:57,014 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:06:57,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:06:57,300 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:06:57,301 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 19:06:57,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 19:06:57.554 UTC [13520] WARNING: there is no transaction in progress
2026-06-20 19:06:57,562 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 19:06:57,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:06:57,842 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 19:06:57,842 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:06:57,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 19:06:58,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 19:06:58,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6wrf.c.gcp-upstream.internal','N') ;"
2026-06-20 19:06:58,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:06:58,914 [I] Table x_portal_user does not exist in database ranger
2026-06-20 19:06:58,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:06:59,182 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 19:06:59,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:06:59,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:06:59,715 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 19:06:59,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 19:07:00.890 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:00.910 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:00.931 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:00.952 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:00.969 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:00.991 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:01.058 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:01.066 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:01.082 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:01.095 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:01.109 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:01.132 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:01.145 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:01.154 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:01.164 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:01.172 UTC [13628] WARNING: there is no transaction in progress
2026-06-20 19:07:02,078 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 19:07:02,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 19:07:02,330 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 19:07:02,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:07:02,598 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 19:07:02,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:07:02,872 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 19:07:02,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:07:03,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:07:03,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6wrf.c.gcp-upstream.internal';"
2026-06-20 19:07:03,709 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 19:07:03,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 19:07:03,979 [I] DB_PATCHES have already been applied
I20260620 19:07:03.985955 1642 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-admin:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8BnTdh/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 19:07:03.986037 1642 mini_ranger.cc:193] Using host: 127.1.154.148
I20260620 19:07:03.990479 1642 mini_ranger.cc:227] Ranger admin URL: http://127.1.154.148:35087
Jun 20, 2026 7:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 19:07:05.137116 1642 mini_ranger.cc:160] Time spent starting Ranger: real 9.612s user 0.000s sys 0.010s
Jun 20, 2026 7:07:26 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:07:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 7:07:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:07:26 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:07:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:07:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:07:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:07:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:07:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:07:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:07:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:07:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:07:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:07:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:07:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:07:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:07:26 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:07:27 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 19:07:28.455104 1642 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.154.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.154.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/rangerkms_127.1.154.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.154.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/rangerkms_127.1.154.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.154.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.154.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.154.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/HTTP_127.1.154.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.154.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/HTTP_127.1.154.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 19:07:28 dist-test-slave-6wrf krb5kdc[13415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982448, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 19:07:28.559355 1642 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 19:07:28.569916 1642 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 19:07:29.819408 1642 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 19:07:29.819590 1642 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-kms
I20260620 19:07:29.819617 1642 mini_ranger_kms.cc:227] Using postgres at 127.1.154.148:35583
2026-06-20 19:07:30,015 [I] DB FLAVOR :POSTGRES
2026-06-20 19:07:30,015 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:07:30,015 [I] Checking connection
2026-06-20 19:07:30,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 19:07:30,294 [I] connection success
2026-06-20 19:07:30,294 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 19:07:30,294 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 19:07:30,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 19:07:30,614 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 19:07:30,614 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:07:30,615 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 19:07:30,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.154.148:35583/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 19:07:30,925 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 19:07:31.483024 1642 mini_ranger_kms.cc:310] Created kms service
I20260620 19:07:31.713392 1642 mini_ranger_kms.cc:326] Created kudu user
I20260620 19:07:31.774441 1642 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 19:07:31 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982448, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
I20260620 19:07:32.222694 1642 mini_ranger_kms.cc:384] Added ranger policy
I20260620 19:07:32.222946 1642 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-kms:/tmp/dist-test-task8BnTdh/build/debug/bin/postgresql.jar:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8BnTdh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 19:07:32.222975 1642 mini_ranger_kms.cc:240] Using host: 127.1.154.148
I20260620 19:07:32.226639 1642 mini_ranger_kms.cc:276] Ranger KMS PID: 13809
I20260620 19:07:32.226722 1642 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.154.148:50011
Jun 20, 2026 7:07:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:07:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:07:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:07:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:07:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:07:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 19:07:34.521620 1642 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.962s user 0.017s sys 0.159s
I20260620 19:07:34.521747 1642 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 19:07:34.521812 1642 mini_ranger_kms.cc:401] 127.1.154.148:50011/kms/v1/keys
Jun 20 19:07:34 dist-test-slave-6wrf krb5kdc[13415](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982454, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.154.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 19:07:35 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982454, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.154.148@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
Jun 20 19:07:35 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982454, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.154.148@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
Jun 20 19:07:38 dist-test-slave-6wrf krb5kdc[13415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:07:38.098941 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:33851
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:33851
--ranger_config_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.154.148:50011/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:38.232076 13859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:38.232431 13859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:38.232514 13859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:38.237188 13859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:07:38.237303 13859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:38.237342 13859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:38.237409 13859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:07:38.237447 13859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:07:38.243665 13859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.154.148:50011/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:33851
--ranger_config_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:33851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:38.245478 13859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:38.247274 13859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:07:38.257889 13859 server_base.cc:1034] running on GCE node
W20260620 19:07:38.257840 13864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:38.257813 13865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:38.258128 13867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:38.258752 13859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:38.259990 13859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:38.261221 13859 hybrid_clock.cc:648] HybridClock initialized: now 1781982458261190 us; error 51 us; skew 500 ppm
Jun 20 19:07:38 dist-test-slave-6wrf krb5kdc[13415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982458, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:38.271634 13859 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:07:38.272866 13859 webserver.cc:460] Webserver started at http://127.1.154.190:46677/ using document root <none> and password file <none>
I20260620 19:07:38.273517 13859 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:38.273581 13859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:38.273928 13859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:07:38 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982458, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
I20260620 19:07:38.333256 13859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "36d5ae91e0c14fc480fbd81fa79535c5"
format_stamp: "Formatted at 2026-06-20 19:07:38 on dist-test-slave-6wrf"
server_key: "f19cf3034967aeda7c44a837e06c62c5"
server_key_iv: "f9e28ad775fb3b8a6955c1965dcdcfa8"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:38.333992 13859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "36d5ae91e0c14fc480fbd81fa79535c5"
format_stamp: "Formatted at 2026-06-20 19:07:38 on dist-test-slave-6wrf"
server_key: "f19cf3034967aeda7c44a837e06c62c5"
server_key_iv: "f9e28ad775fb3b8a6955c1965dcdcfa8"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:38.338085 13859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:07:38.354110 13874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:38.355592 13859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:07:38.355769 13859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "36d5ae91e0c14fc480fbd81fa79535c5"
format_stamp: "Formatted at 2026-06-20 19:07:38 on dist-test-slave-6wrf"
server_key: "f19cf3034967aeda7c44a837e06c62c5"
server_key_iv: "f9e28ad775fb3b8a6955c1965dcdcfa8"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:38.355942 13859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:38.377280 13859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:38.380645 13859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:38.380873 13859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:38.389266 13859 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:33851
I20260620 19:07:38.389274 13926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:33851 every 8 connection(s)
I20260620 19:07:38.390278 13859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:07:38.392514 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 13859
I20260620 19:07:38.392634 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:07:38.392927 1642 external_mini_cluster.cc:1428] Setting key dbb6d929634d84f0566e821dca4648ef
I20260620 19:07:38.393020 13927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 19:07:38 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:38.402554 13927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5: Bootstrap starting.
I20260620 19:07:38.406041 13927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:38.407476 13927 log.cc:824] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:38.408841 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:38.394508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.1.154.190:33851)
0620 19:07:38.394906 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:38.394930 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:38.394965 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:07:38.395569 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:38.395583 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:38.395889 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:38.396230 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:38.397043 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:38.398156 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:38.399017 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:38.399481 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:38.399655 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:38.403170 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:38.403176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:38.403191 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:38.403249 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:38.405086 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:38.405633 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:38.405640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:38.405646 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:38.405771 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:38.406101 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:38.406106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:38.406185 (+ 79us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:38.406634 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:38.406862 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:38.407180 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 19:07:38.407508 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":142,"threads_started":1}
I20260620 19:07:38.410396 13927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5: No bootstrap required, opened a new log
I20260620 19:07:38.415527 13927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 33851 } }
I20260620 19:07:38.415983 13927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:38.416062 13927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d5ae91e0c14fc480fbd81fa79535c5, State: Initialized, Role: FOLLOWER
I20260620 19:07:38.417016 13927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 33851 } }
I20260620 19:07:38.417285 13927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:07:38.417421 13927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:07:38.417526 13927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:38.418933 13927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 33851 } }
I20260620 19:07:38.419368 13927 leader_election.cc:302] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36d5ae91e0c14fc480fbd81fa79535c5; no voters:
I20260620 19:07:38.419778 13927 leader_election.cc:288] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:07:38.420295 13932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:38.420522 13932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 1 LEADER]: Becoming Leader. State: Replica: 36d5ae91e0c14fc480fbd81fa79535c5, State: Running, Role: LEADER
I20260620 19:07:38.420850 13932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 33851 } }
I20260620 19:07:38.421422 13927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:07:38.422106 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 33851 } } }
I20260620 19:07:38.422402 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:38.422308 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36d5ae91e0c14fc480fbd81fa79535c5. Latest consensus state: current_term: 1 leader_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 33851 } } }
I20260620 19:07:38.422706 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:38.423683 13927 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:07:38.423946 13941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:07:38.431097 13941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:07:38.440999 13941 catalog_manager.cc:1347] Generated new cluster ID: 8918403c0fe64b8cac0994f5f8190aa0
I20260620 19:07:38.441098 13941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:07:38.486655 13941 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:07:38.488267 13941 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:07:38.504341 13941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5: Generated new TSK 0
I20260620 19:07:38.505409 13941 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:07:39.274473 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.154.148:50011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:33851
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:39.438827 13972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:39.439174 13972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:39.439244 13972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:39.445819 13972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:39.445910 13972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:39.445998 13972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:07:39.455015 13972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.154.148:50011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:33851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:39.456303 13972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:39.457391 13972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:39.470270 13982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:39.470465 13979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:39.470316 13980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:39.471329 13972 server_base.cc:1034] running on GCE node
I20260620 19:07:39.471887 13972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:39.472833 13972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:39.474045 13972 hybrid_clock.cc:648] HybridClock initialized: now 1781982459474025 us; error 34 us; skew 500 ppm
Jun 20 19:07:39 dist-test-slave-6wrf krb5kdc[13415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:39.477784 13972 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:07:39.478688 13972 webserver.cc:460] Webserver started at http://127.1.154.129:46007/ using document root <none> and password file <none>
I20260620 19:07:39.479272 13972 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:39.479362 13972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:39.479570 13972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:07:39 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
I20260620 19:07:39.497709 13972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "9283df07ab1148ffb4c378c504227818"
format_stamp: "Formatted at 2026-06-20 19:07:39 on dist-test-slave-6wrf"
server_key: "ec44e30d82caf94b57b7c777074867fb"
server_key_iv: "474b243528949a9fee7aa3d21983a743"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:39.498556 13972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "9283df07ab1148ffb4c378c504227818"
format_stamp: "Formatted at 2026-06-20 19:07:39 on dist-test-slave-6wrf"
server_key: "ec44e30d82caf94b57b7c777074867fb"
server_key_iv: "474b243528949a9fee7aa3d21983a743"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:39.504271 13972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 19:07:39.536900 13989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:39.541045 13972 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.001s
I20260620 19:07:39.541239 13972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "9283df07ab1148ffb4c378c504227818"
format_stamp: "Formatted at 2026-06-20 19:07:39 on dist-test-slave-6wrf"
server_key: "ec44e30d82caf94b57b7c777074867fb"
server_key_iv: "474b243528949a9fee7aa3d21983a743"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:39.541400 13972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:39.567921 13972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:39.571926 13972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:39.572216 13972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:39.574270 13972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:39.575737 13972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:39.575852 13972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:39.575984 13972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:39.576069 13972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:39.591677 13972 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:32873
I20260620 19:07:39.591920 14104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:32873 every 8 connection(s)
I20260620 19:07:39.592985 13972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:07:39.595134 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 13972
I20260620 19:07:39.595242 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:07:39.595542 1642 external_mini_cluster.cc:1428] Setting key c66ec927a8e0d3617d9ded5d2d624dd1
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
Jun 20 19:07:39 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:39.620147 14107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:39.595684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:33851 (local address 127.1.154.129:37625)
0620 19:07:39.596264 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:39.596328 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:39.597327 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:39.601489 (+ 4162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:39.601509 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:39.602165 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:39.603276 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:39.603314 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:39.605235 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:39.605242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:39.607151 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:39.607177 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:39.608070 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:39.608075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:39.608375 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:39.609495 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:39.609526 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:39.613554 (+ 4028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:39.616436 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:39.616444 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:39.616466 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:39.616887 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:39.617222 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:39.617225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:39.617227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:39.617426 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:39.618024 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:39.618033 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:39.618423 (+ 390us) client_negotiation.cc:769] Sending connection context
0620 19:07:39.618634 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 19:07:39.619015 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":131,"threads_started":1}
I20260620 19:07:39.621137 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:39.597003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:37625 (local address 127.1.154.190:33851)
0620 19:07:39.597756 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:39.597762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:39.597775 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:07:39.597821 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:39.597824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:39.597875 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:39.603652 (+ 5777us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:39.603851 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:39.604842 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:39.607406 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:39.607819 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:39.610716 (+ 2897us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:39.613910 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:39.613914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:39.613918 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:39.613958 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:39.616256 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:39.617011 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:39.617015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:39.617017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:39.617105 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:39.617546 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:39.617550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:39.617552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:39.617860 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:39.617964 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:39.620810 (+ 2846us) server_negotiation.cc:299] Negotiation successful
0620 19:07:39.620958 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":86,"threads_started":1}
I20260620 19:07:39.623118 14105 heartbeater.cc:347] Connected to a master server at 127.1.154.190:33851
I20260620 19:07:39.623392 14105 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:39.624001 14105 heartbeater.cc:511] Master 127.1.154.190:33851 requested a full tablet report, sending...
I20260620 19:07:39.626017 13891 ts_manager.cc:194] Registered new tserver with Master: 9283df07ab1148ffb4c378c504227818 (127.1.154.129:32873)
I20260620 19:07:39.627599 13891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:37625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:07:39.637341 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.154.148:50011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:33851
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:07:39 dist-test-slave-6wrf krb5kdc[13415](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 19:07:39.798815 14111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:39.799207 14111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:39.799274 14111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:39.805505 14111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:39.805647 14111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:39.805778 14111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:07:39.813560 14111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.154.148:50011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:33851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:39.814899 14111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:39.816169 14111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:39.828763 14117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:39.828791 14118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:39.828755 14120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:39.829481 14111 server_base.cc:1034] running on GCE node
I20260620 19:07:39.829991 14111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:39.830818 14111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:39.832041 14111 hybrid_clock.cc:648] HybridClock initialized: now 1781982459832020 us; error 35 us; skew 500 ppm
Jun 20 19:07:39 dist-test-slave-6wrf krb5kdc[13415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:39.836440 14111 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:07:39.838238 14111 webserver.cc:460] Webserver started at http://127.1.154.130:39631/ using document root <none> and password file <none>
I20260620 19:07:39.839186 14111 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:39.839283 14111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:39.839589 14111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:07:39 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
I20260620 19:07:39.865255 14111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "a8f0b9af2fbb49d1a03287ed375a3543"
format_stamp: "Formatted at 2026-06-20 19:07:39 on dist-test-slave-6wrf"
server_key: "ce483ba7c8e7d8f279dc6efeaae4f52e"
server_key_iv: "518f27cd6bedbab749d72b8d275288eb"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:39.866007 14111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "a8f0b9af2fbb49d1a03287ed375a3543"
format_stamp: "Formatted at 2026-06-20 19:07:39 on dist-test-slave-6wrf"
server_key: "ce483ba7c8e7d8f279dc6efeaae4f52e"
server_key_iv: "518f27cd6bedbab749d72b8d275288eb"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:39.870999 14111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 19:07:39.886456 14127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:39.888100 14111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:07:39.888238 14111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "a8f0b9af2fbb49d1a03287ed375a3543"
format_stamp: "Formatted at 2026-06-20 19:07:39 on dist-test-slave-6wrf"
server_key: "ce483ba7c8e7d8f279dc6efeaae4f52e"
server_key_iv: "518f27cd6bedbab749d72b8d275288eb"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:39.888381 14111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:39.909842 14111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:39.917713 14111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:39.917910 14111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:39.918589 14111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:39.919657 14111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:39.919723 14111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:39.919785 14111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:39.919818 14111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:39.932072 14111 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:32867
I20260620 19:07:39.933256 14111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:07:39.939587 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 14111
I20260620 19:07:39.939738 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:07:39.940045 1642 external_mini_cluster.cc:1428] Setting key e462118de2cdf2d853f644d480cedf04
I20260620 19:07:39.946779 14240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:32867 every 8 connection(s)
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
Jun 20 19:07:39 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:39.976567 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:39.955974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:54213 (local address 127.1.154.190:33851)
0620 19:07:39.956115 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:39.956120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:39.956528 (+ 408us) server_negotiation.cc:407] Connection header received
0620 19:07:39.957970 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:39.957973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:39.958030 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:39.958090 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:39.962345 (+ 4255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:39.963179 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:39.964930 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:39.965303 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:39.965388 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:39.971752 (+ 6364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:39.971757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:39.971760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:39.971804 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:39.973671 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:39.974571 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:39.974575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:39.974577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:39.974671 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:39.975111 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:39.975115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:39.975117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:39.975400 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:39.975491 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:39.976240 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 19:07:39.976393 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:07:39.978627 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:39.955449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:33851 (local address 127.1.154.130:54213)
0620 19:07:39.956375 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:39.956428 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:39.957663 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:39.960923 (+ 3260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:39.960935 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:39.961318 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:39.962129 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:39.962151 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:39.963357 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:39.963360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:39.964783 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:39.964794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:39.965661 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:39.965667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:39.965868 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:39.969186 (+ 3318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:39.969210 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:39.971481 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:39.973857 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:39.973864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:39.973884 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:39.974431 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:39.974786 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:39.974789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:39.974791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:39.974995 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:39.975565 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:39.975572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:39.975941 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 19:07:39.977085 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0620 19:07:39.977458 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":154,"threads_started":1}
I20260620 19:07:39.983121 14241 heartbeater.cc:347] Connected to a master server at 127.1.154.190:33851
I20260620 19:07:39.983393 14241 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:39.984580 14241 heartbeater.cc:511] Master 127.1.154.190:33851 requested a full tablet report, sending...
I20260620 19:07:39.986472 13891 ts_manager.cc:194] Registered new tserver with Master: a8f0b9af2fbb49d1a03287ed375a3543 (127.1.154.130:32867)
I20260620 19:07:39.987715 13891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:54213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:07:39.999193 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.154.148:50011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:33851
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:07:40 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
W20260620 19:07:40.171312 14246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:40.171679 14246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:40.171743 14246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:40.178651 14246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:40.178743 14246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:40.178841 14246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:07:40.187140 14246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.154.148:50011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:33851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:40.188596 14246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:40.189926 14246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:40.202802 14252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:40.202981 14251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:40.203110 14254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:40.203423 14246 server_base.cc:1034] running on GCE node
I20260620 19:07:40.204180 14246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:40.204955 14246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:40.206104 14246 hybrid_clock.cc:648] HybridClock initialized: now 1781982460206038 us; error 78 us; skew 500 ppm
Jun 20 19:07:40 dist-test-slave-6wrf krb5kdc[13415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982460, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:40.209524 14246 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:07:40.210867 14246 webserver.cc:460] Webserver started at http://127.1.154.131:34443/ using document root <none> and password file <none>
I20260620 19:07:40.211561 14246 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:40.211771 14246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:40.212024 14246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:07:40 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982460, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
I20260620 19:07:40.229353 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "b2076922b46d4fda948438069871351c"
format_stamp: "Formatted at 2026-06-20 19:07:40 on dist-test-slave-6wrf"
server_key: "8167f6a299a83b215fd22768b6e593e8"
server_key_iv: "6b90ae2c8f5ed96349e5cc73ab262bac"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:40.238114 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "b2076922b46d4fda948438069871351c"
format_stamp: "Formatted at 2026-06-20 19:07:40 on dist-test-slave-6wrf"
server_key: "8167f6a299a83b215fd22768b6e593e8"
server_key_iv: "6b90ae2c8f5ed96349e5cc73ab262bac"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:40.244119 14246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 19:07:40.272382 14261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:40.274108 14246 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 19:07:40.274238 14246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "b2076922b46d4fda948438069871351c"
format_stamp: "Formatted at 2026-06-20 19:07:40 on dist-test-slave-6wrf"
server_key: "8167f6a299a83b215fd22768b6e593e8"
server_key_iv: "6b90ae2c8f5ed96349e5cc73ab262bac"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:40.274642 14246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:40.299650 14246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:40.303406 14246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:40.303647 14246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:40.304507 14246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:40.305620 14246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:40.305694 14246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:40.305760 14246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:40.305790 14246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:40.316854 14246 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:33911
I20260620 19:07:40.316934 14374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:33911 every 8 connection(s)
I20260620 19:07:40.318070 14246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:07:40.320664 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 14246
I20260620 19:07:40.320796 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:07:40.321099 1642 external_mini_cluster.cc:1428] Setting key ab4ddc88b382110b75f80d429ccfb9c2
Jun 20 19:07:40 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982460, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:40.335494 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:40.321768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:42349 (local address 127.1.154.190:33851)
0620 19:07:40.321910 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:40.321914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:40.322330 (+ 416us) server_negotiation.cc:407] Connection header received
0620 19:07:40.323588 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:40.323591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:40.323670 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:40.323718 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:40.325293 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:40.326158 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:40.327831 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:40.328169 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:40.328242 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:40.331238 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:40.331242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:40.331245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:40.331272 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:40.333026 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:40.333667 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:40.333670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:40.333672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:40.333755 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:40.334176 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:40.334179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:40.334181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:40.334408 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:40.334491 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:40.335197 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 19:07:40.335329 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:07:40.337062 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:40.320655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:33851 (local address 127.1.154.131:42349)
0620 19:07:40.322188 (+ 1533us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:40.322243 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:40.323363 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:40.324129 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:40.324138 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:40.324480 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:40.325130 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:40.325150 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:40.326316 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:40.326319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:40.327694 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:40.327705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:40.328441 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:40.328445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:40.328644 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:40.329363 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:40.329384 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:40.331047 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:40.333167 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:40.333173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:40.333190 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:40.333520 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:40.333867 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:40.333870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:40.333872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:40.334065 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:40.334559 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:40.334566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:40.334919 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 19:07:40.335702 (+ 783us) client_negotiation.cc:240] Negotiation successful
0620 19:07:40.336035 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1315,"thread_start_us":133,"threads_started":1}
I20260620 19:07:40.338402 14375 heartbeater.cc:347] Connected to a master server at 127.1.154.190:33851
I20260620 19:07:40.338641 14375 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:40.339174 14375 heartbeater.cc:511] Master 127.1.154.190:33851 requested a full tablet report, sending...
I20260620 19:07:40.340233 13891 ts_manager.cc:194] Registered new tserver with Master: b2076922b46d4fda948438069871351c (127.1.154.131:33911)
I20260620 19:07:40.340905 13891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:42349
I20260620 19:07:40.347698 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:07:40.348762 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 13972
I20260620 19:07:40.358196 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 14111
I20260620 19:07:40.364516 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 14246
I20260620 19:07:40.372051 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 13859
I20260620 19:07:40.378465 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:33851
--webserver_interface=127.1.154.190
--webserver_port=46677
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:33851
--ranger_config_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.154.148:50011/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.154.148:50011/kms with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:40.504030 14378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:40.504321 14378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:40.504364 14378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:40.509202 14378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:07:40.509289 14378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:40.509310 14378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:40.509328 14378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:07:40.509344 14378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:07:40.515851 14378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.154.148:50011/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:33851
--ranger_config_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:33851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=46677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:40.516950 14378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:40.518344 14378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:40.527815 14384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:40.527925 14386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:40.527854 14383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:40.528339 14378 server_base.cc:1034] running on GCE node
I20260620 19:07:40.529129 14378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:40.530701 14378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:40.531952 14378 hybrid_clock.cc:648] HybridClock initialized: now 1781982460531896 us; error 69 us; skew 500 ppm
Jun 20 19:07:40 dist-test-slave-6wrf krb5kdc[13415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982460, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:40.535481 14378 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:07:40.536471 14378 webserver.cc:460] Webserver started at http://127.1.154.190:46677/ using document root <none> and password file <none>
I20260620 19:07:40.537060 14378 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:40.537163 14378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:07:40 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982460, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
I20260620 19:07:40.591012 14378 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:07:40.594070 14394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:40.596290 14378 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:07:40.596629 14378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "36d5ae91e0c14fc480fbd81fa79535c5"
format_stamp: "Formatted at 2026-06-20 19:07:38 on dist-test-slave-6wrf"
server_key: "f19cf3034967aeda7c44a837e06c62c5"
server_key_iv: "f9e28ad775fb3b8a6955c1965dcdcfa8"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:40.597537 14378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:40.634234 14378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:40.638447 14378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:40.638739 14378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:40.648124 14378 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:33851
I20260620 19:07:40.648126 14446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:33851 every 8 connection(s)
I20260620 19:07:40.650153 14378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:07:40.654690 14447 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:07:40.656440 14447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5: Bootstrap starting.
I20260620 19:07:40.658193 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 14378
I20260620 19:07:40.658676 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.154.148:50011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:32873
--local_ip_for_outbound_sockets=127.1.154.129
--tserver_master_addrs=127.1.154.190:33851
--webserver_port=46007
--webserver_interface=127.1.154.129
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.154.148:50011/kms with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:07:40.676445 14447 log.cc:824] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:40.683295 14447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:07:40.683729 14447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5: Bootstrap complete.
I20260620 19:07:40.687907 14447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 33851 } }
I20260620 19:07:40.688298 14447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d5ae91e0c14fc480fbd81fa79535c5, State: Initialized, Role: FOLLOWER
I20260620 19:07:40.688820 14447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 33851 } }
I20260620 19:07:40.688934 14447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:07:40.689007 14447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:07:40.689100 14447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:07:40.691073 14447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 33851 } }
I20260620 19:07:40.691429 14447 leader_election.cc:302] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36d5ae91e0c14fc480fbd81fa79535c5; no voters:
I20260620 19:07:40.691768 14447 leader_election.cc:288] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:07:40.691852 14451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:07:40.692054 14451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [term 2 LEADER]: Becoming Leader. State: Replica: 36d5ae91e0c14fc480fbd81fa79535c5, State: Running, Role: LEADER
I20260620 19:07:40.692348 14451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 33851 } }
I20260620 19:07:40.692973 14447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:07:40.693805 14452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 33851 } } }
I20260620 19:07:40.693948 14452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:40.694895 14453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36d5ae91e0c14fc480fbd81fa79535c5. Latest consensus state: current_term: 2 leader_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d5ae91e0c14fc480fbd81fa79535c5" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 33851 } } }
I20260620 19:07:40.695091 14453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:40.695505 14460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:07:40.695961 14447 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:07:40.702169 14460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:07:40.703341 14460 catalog_manager.cc:1259] Loaded cluster ID: 8918403c0fe64b8cac0994f5f8190aa0
I20260620 19:07:40.703414 14460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:07:40.706357 14460 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:07:40.707062 14460 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5: Loaded TSK: 0
I20260620 19:07:40.709092 14460 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:07:40.814134 14449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:40.814494 14449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:40.814559 14449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:40.821653 14449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:40.821748 14449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:40.821844 14449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:07:40.832633 14449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.154.148:50011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:32873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=46007
--tserver_master_addrs=127.1.154.190:33851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:40.834038 14449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:40.835336 14449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:40.848894 14485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:40.848552 14484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:40.849426 14449 server_base.cc:1034] running on GCE node
W20260620 19:07:40.848536 14487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:40.850124 14449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:40.850960 14449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:40.852159 14449 hybrid_clock.cc:648] HybridClock initialized: now 1781982460852133 us; error 44 us; skew 500 ppm
Jun 20 19:07:40 dist-test-slave-6wrf krb5kdc[13415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982460, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:40.856945 14449 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:07:40.858345 14449 webserver.cc:460] Webserver started at http://127.1.154.129:46007/ using document root <none> and password file <none>
I20260620 19:07:40.859143 14449 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:40.859266 14449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:07:40 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982460, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
I20260620 19:07:40.905041 14449 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.000s sys 0.003s
I20260620 19:07:40.918888 14495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:40.921535 14449 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.002s sys 0.000s
I20260620 19:07:40.921751 14449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "9283df07ab1148ffb4c378c504227818"
format_stamp: "Formatted at 2026-06-20 19:07:39 on dist-test-slave-6wrf"
server_key: "ec44e30d82caf94b57b7c777074867fb"
server_key_iv: "474b243528949a9fee7aa3d21983a743"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:40.922573 14449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:40.958957 14449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:40.962932 14449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:40.963392 14449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:40.964193 14449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:40.965422 14449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:40.965505 14449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:40.965566 14449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:40.965631 14449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 19:07:40.978644 14449 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:32873
I20260620 19:07:40.979756 14449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:07:40.981426 14608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:32873 every 8 connection(s)
I20260620 19:07:40.986627 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 14449
I20260620 19:07:40.987107 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.154.148:50011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:32867
--local_ip_for_outbound_sockets=127.1.154.130
--tserver_master_addrs=127.1.154.190:33851
--webserver_port=39631
--webserver_interface=127.1.154.130
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.154.148:50011/kms with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:07:40 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982460, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:41.010418 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:40.983222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:33851 (local address 127.1.154.129:52553)
0620 19:07:40.983959 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:40.984041 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:40.985491 (+ 1450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:40.988987 (+ 3496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:40.988999 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:40.989419 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:40.990172 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:40.990190 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:40.991690 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:40.991694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:40.993737 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:40.993791 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:40.995896 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:40.995901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:40.996397 (+ 496us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:40.997424 (+ 1027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:40.997450 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:40.999894 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:41.004735 (+ 4841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:41.004745 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:41.004771 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:41.005485 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:41.006176 (+ 691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:41.006180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:41.006182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:41.006403 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:41.007524 (+ 1121us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:41.007533 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:41.008275 (+ 742us) client_negotiation.cc:769] Sending connection context
0620 19:07:41.008789 (+ 514us) client_negotiation.cc:240] Negotiation successful
0620 19:07:41.009234 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":155,"threads_started":1}
I20260620 19:07:41.012161 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:40.983172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:52553 (local address 127.1.154.190:33851)
0620 19:07:40.983998 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:40.984028 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:40.984220 (+ 192us) server_negotiation.cc:407] Connection header received
0620 19:07:40.986305 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:40.986333 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:40.986934 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:40.987731 (+ 797us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:40.990413 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:40.991426 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:40.994218 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:40.995472 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:40.995682 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:41.000359 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:41.000367 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:41.000389 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:41.000434 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:41.004419 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:41.005747 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:41.005757 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:41.005765 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:41.006003 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:41.006545 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:41.006549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:41.006551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:41.007232 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:41.007450 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:41.010928 (+ 3478us) server_negotiation.cc:299] Negotiation successful
0620 19:07:41.011269 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":267,"threads_started":1}
I20260620 19:07:41.016709 14609 heartbeater.cc:347] Connected to a master server at 127.1.154.190:33851
I20260620 19:07:41.017011 14609 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:41.017848 14609 heartbeater.cc:511] Master 127.1.154.190:33851 requested a full tablet report, sending...
W20260620 19:07:41.022116 14609 heartbeater.cc:644] Failed to heartbeat to 127.1.154.190:33851 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:07:41.022346 14609 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:41.022848 14609 heartbeater.cc:511] Master 127.1.154.190:33851 requested a full tablet report, sending...
I20260620 19:07:41.025053 14609 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:41.025756 14609 heartbeater.cc:511] Master 127.1.154.190:33851 requested a full tablet report, sending...
W20260620 19:07:41.027350 14609 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:07:41.172466 14613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:41.173206 14613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:41.173360 14613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:41.180377 14613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:41.180500 14613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:41.180598 14613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:07:41.189954 14613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.154.148:50011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:32867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=39631
--tserver_master_addrs=127.1.154.190:33851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:41.191552 14613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:41.192734 14613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:41.209579 14619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:41.209558 14618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:41.210247 14621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:41.211021 14613 server_base.cc:1034] running on GCE node
I20260620 19:07:41.211714 14613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:41.212577 14613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:41.213807 14613 hybrid_clock.cc:648] HybridClock initialized: now 1781982461213741 us; error 89 us; skew 500 ppm
Jun 20 19:07:41 dist-test-slave-6wrf krb5kdc[13415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:41.216984 14613 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:07:41.218143 14613 webserver.cc:460] Webserver started at http://127.1.154.130:39631/ using document root <none> and password file <none>
I20260620 19:07:41.218856 14613 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:41.218972 14613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:07:41 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
I20260620 19:07:41.255213 14613 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 19:07:41.259764 14629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:41.261631 14613 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:07:41.261847 14613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "a8f0b9af2fbb49d1a03287ed375a3543"
format_stamp: "Formatted at 2026-06-20 19:07:39 on dist-test-slave-6wrf"
server_key: "ce483ba7c8e7d8f279dc6efeaae4f52e"
server_key_iv: "518f27cd6bedbab749d72b8d275288eb"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:41.262748 14613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:41.298470 14613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:41.303280 14613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:41.303507 14613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:41.304204 14613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:41.305061 14613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:41.305109 14613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:41.305181 14613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:41.305217 14613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:41.319784 14613 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:32867
I20260620 19:07:41.319871 14743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:32867 every 8 connection(s)
I20260620 19:07:41.321751 14613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:07:41.329797 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 14613
I20260620 19:07:41.330482 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.154.148:50011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:33911
--local_ip_for_outbound_sockets=127.1.154.131
--tserver_master_addrs=127.1.154.190:33851
--webserver_port=34443
--webserver_interface=127.1.154.131
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.154.148:50011/kms with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:07:41 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:41.343441 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:41.324485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:52889 (local address 127.1.154.190:33851)
0620 19:07:41.324664 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:41.324669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:41.325677 (+ 1008us) server_negotiation.cc:407] Connection header received
0620 19:07:41.327166 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:41.327170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:41.327233 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:41.327342 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:41.330321 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:41.331235 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:41.333833 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:41.334135 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:41.334261 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:41.337988 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:41.337991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:41.337993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:41.338191 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:41.340719 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:41.341660 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:41.341665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:41.341672 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:41.341766 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:41.342169 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:41.342172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:41.342174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:41.342387 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:41.342504 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:41.343090 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 19:07:41.343254 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:07:41.345731 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:41.324979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:33851 (local address 127.1.154.130:52889)
0620 19:07:41.325484 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:41.325539 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:41.326935 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:41.327518 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:41.327525 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:41.328662 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:41.329877 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:41.330036 (+ 159us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:41.331810 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:41.331817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:41.333491 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:41.333516 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:41.334334 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:41.334340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:41.334596 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:41.335409 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:41.335433 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:41.337489 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:41.341052 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:41.341059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:41.341107 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:41.341494 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:41.341905 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:41.341907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:41.341909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:41.342057 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:41.342502 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:41.342510 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:41.342826 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 19:07:41.343028 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 19:07:41.343379 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":155,"threads_started":1}
I20260620 19:07:41.347353 14744 heartbeater.cc:347] Connected to a master server at 127.1.154.190:33851
I20260620 19:07:41.347612 14744 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:41.348178 14744 heartbeater.cc:511] Master 127.1.154.190:33851 requested a full tablet report, sending...
W20260620 19:07:41.350903 14744 heartbeater.cc:644] Failed to heartbeat to 127.1.154.190:33851 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:07:41.351131 14744 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:41.351404 14744 heartbeater.cc:511] Master 127.1.154.190:33851 requested a full tablet report, sending...
I20260620 19:07:41.352921 14744 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:41.353229 14744 heartbeater.cc:511] Master 127.1.154.190:33851 requested a full tablet report, sending...
W20260620 19:07:41.354638 14744 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:07:41.519870 14747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:41.520224 14747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:41.520303 14747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:41.530630 14747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:41.530753 14747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:41.530857 14747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:07:41.543874 14747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.154.148:50011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:33911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=34443
--tserver_master_addrs=127.1.154.190:33851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:41.545226 14747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:41.546876 14747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:41.564102 14755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:41.564781 14752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:41.565073 14753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:41.565332 14747 server_base.cc:1034] running on GCE node
I20260620 19:07:41.566260 14747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:41.567148 14747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:41.568706 14747 hybrid_clock.cc:648] HybridClock initialized: now 1781982461568631 us; error 94 us; skew 500 ppm
Jun 20 19:07:41 dist-test-slave-6wrf krb5kdc[13415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:41.573668 14747 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:07:41.575196 14747 webserver.cc:460] Webserver started at http://127.1.154.131:34443/ using document root <none> and password file <none>
I20260620 19:07:41.575954 14747 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:41.576072 14747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:07:41 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for HTTP/127.1.154.148@KRBTEST.COM
I20260620 19:07:41.602166 14747 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 19:07:41.606171 14763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:41.608448 14747 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 19:07:41.608623 14747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "b2076922b46d4fda948438069871351c"
format_stamp: "Formatted at 2026-06-20 19:07:40 on dist-test-slave-6wrf"
server_key: "8167f6a299a83b215fd22768b6e593e8"
server_key_iv: "6b90ae2c8f5ed96349e5cc73ab262bac"
server_key_version: "kuduclusterkey@0"
I20260620 19:07:41.609426 14747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:41.652670 14747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:41.658167 14747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:41.658427 14747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:41.659231 14747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:41.660236 14747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:41.660310 14747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:41.660377 14747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:41.660424 14747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:41.673326 14747 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:33911
I20260620 19:07:41.673391 14883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:33911 every 8 connection(s)
I20260620 19:07:41.674536 14747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:07:41.678097 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 14747
Jun 20 19:07:41 dist-test-slave-6wrf krb5kdc[13415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:41.692225 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:41.679819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.1.154.190:33851)
0620 19:07:41.680260 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:41.680271 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:41.680286 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:07:41.680317 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:41.680319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:41.680350 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:41.680441 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:41.682050 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:41.683293 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:41.684789 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:41.685152 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:41.687683 (+ 2531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:41.687824 (+ 141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:41.687827 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:41.687830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:41.687850 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:41.690260 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:41.690806 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:41.690810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:41.690812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:41.690909 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:41.691234 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:41.691238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:41.691240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:41.691478 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:41.691735 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:41.691875 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 19:07:41.692004 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":257,"threads_started":1}
I20260620 19:07:41.694682 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:41.676785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:44487 (local address 127.1.154.190:33851)
0620 19:07:41.676934 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:41.676939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:41.678337 (+ 1398us) server_negotiation.cc:407] Connection header received
0620 19:07:41.680051 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:41.680056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:41.680122 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:41.680183 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:41.682512 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:41.683325 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:41.685662 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:41.686029 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:41.686122 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:41.689420 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:41.689424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:41.689427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:41.689478 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:41.691664 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:41.692627 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:41.692631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:41.692633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:41.692726 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:41.693192 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:41.693197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:41.693198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:41.693458 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:41.693588 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:41.694346 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 19:07:41.694483 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 19:07:41.696307 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:41.677168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:33851 (local address 127.1.154.131:44487)
0620 19:07:41.677971 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:41.678084 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:41.679712 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:41.680687 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:41.680695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:41.681010 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:41.682262 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:41.682295 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:41.683775 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:41.683778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:41.685441 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:41.685455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:41.686347 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:41.686351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:41.686572 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:41.687284 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:41.687306 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:41.689198 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:41.692045 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:41.692052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:41.692071 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:41.692471 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:41.692855 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:41.692873 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:41.692875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:41.693085 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:41.693651 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:41.693659 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:41.694033 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 19:07:41.694872 (+ 839us) client_negotiation.cc:240] Negotiation successful
0620 19:07:41.695261 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":149,"threads_started":1}
I20260620 19:07:41.697520 14884 heartbeater.cc:347] Connected to a master server at 127.1.154.190:33851
I20260620 19:07:41.697822 14884 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:41.698467 14884 heartbeater.cc:511] Master 127.1.154.190:33851 requested a full tablet report, sending...
I20260620 19:07:41.700166 14411 ts_manager.cc:194] Registered new tserver with Master: b2076922b46d4fda948438069871351c (127.1.154.131:33911)
I20260620 19:07:41.701836 14411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:44487
I20260620 19:07:42.028151 14609 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:42.028751 14609 heartbeater.cc:511] Master 127.1.154.190:33851 requested a full tablet report, sending...
I20260620 19:07:42.030055 14409 ts_manager.cc:194] Registered new tserver with Master: 9283df07ab1148ffb4c378c504227818 (127.1.154.129:32873)
I20260620 19:07:42.030745 14409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:52553
Jun 20 19:07:42 dist-test-slave-6wrf krb5kdc[13415](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982462, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:42.355060 14744 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:42.355434 14744 heartbeater.cc:511] Master 127.1.154.190:33851 requested a full tablet report, sending...
I20260620 19:07:42.356369 14409 ts_manager.cc:194] Registered new tserver with Master: a8f0b9af2fbb49d1a03287ed375a3543 (127.1.154.130:32867)
I20260620 19:07:42.357213 14409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:52889
I20260620 19:07:42.367863 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:07:42.385861 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:42.369719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.1.154.190:33851)
0620 19:07:42.374035 (+ 4316us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:42.374040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:42.374062 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:07:42.374119 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:42.374122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:42.374177 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:42.374308 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:42.375754 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.376646 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.377916 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.378279 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.378389 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:42.381080 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:42.381084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:42.381087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:42.381119 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:42.383312 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:42.383893 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:42.383900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:42.383903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:42.384009 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:42.384362 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:42.384366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:42.384367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:42.384639 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:42.384776 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:42.385486 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 19:07:42.385667 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4188,"thread_start_us":99,"threads_started":1}
I20260620 19:07:42.392287 14409 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:07:42.398011 14409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:07:42.427168 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:42.418339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:33911 (local address 127.0.0.1:32798)
0620 19:07:42.418802 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:42.418857 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:42.418991 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:42.421215 (+ 2224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:42.421222 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:42.421254 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:42.421635 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:42.421649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.423655 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.423661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:42.425293 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:42.425309 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.426557 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.426560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:42.426653 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:42.426687 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 19:07:42.426732 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:07:42.426986 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":167,"thread_start_us":130,"threads_started":1}
I20260620 19:07:42.427464 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:42.420772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:32873 (local address 127.0.0.1:57772)
0620 19:07:42.421118 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:42.421139 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:42.421252 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:42.422163 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:42.422166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:42.422185 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:42.422314 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:42.422322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.423904 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.423908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:42.425213 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:42.425228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.426337 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.426340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:42.426444 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:42.426490 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 19:07:42.426581 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 19:07:42.427055 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":1,"thread_start_us":136,"threads_started":1}
I20260620 19:07:42.427827 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:42.420128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.1.154.131:33911)
0620 19:07:42.420436 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:42.420445 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:42.420475 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:07:42.420580 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:42.420586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:42.420998 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:42.422033 (+ 1035us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:42.422238 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.423420 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.425481 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.426406 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.426880 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:42.427514 (+ 634us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:42.427571 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 19:07:42.427684 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
I20260620 19:07:42.432543 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:42.419567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:32867 (local address 127.0.0.1:58798)
0620 19:07:42.419996 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:42.420016 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:42.420155 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:42.426488 (+ 6333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:42.426494 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:42.426537 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:42.426634 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:42.426640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.429241 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.429244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:42.430526 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:42.430537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.431788 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.431792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:42.431953 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:42.432151 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 19:07:42.432295 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 19:07:42.432408 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":201,"threads_started":1}
I20260620 19:07:42.432569 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:42.419783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.1.154.130:32867)
0620 19:07:42.422784 (+ 3001us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:42.422791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:42.422851 (+ 60us) server_negotiation.cc:407] Connection header received
0620 19:07:42.422944 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:42.422949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:42.423129 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:42.426794 (+ 3665us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:42.426953 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.428327 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.430691 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.431645 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.431768 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:42.431913 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:42.432345 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 19:07:42.432443 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2332,"thread_start_us":108,"threads_started":1}
I20260620 19:07:42.431743 14804 tablet_service.cc:1467] Processing CreateTablet for tablet d6d4660930d1469985241d87717d7524 (DEFAULT_TABLE table=test-table [id=a790eb8a10a64b409b7f9445ff49f3ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:42.433707 14804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6d4660930d1469985241d87717d7524. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:42.436005 14675 tablet_service.cc:1467] Processing CreateTablet for tablet d6d4660930d1469985241d87717d7524 (DEFAULT_TABLE table=test-table [id=a790eb8a10a64b409b7f9445ff49f3ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:42.437077 14675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6d4660930d1469985241d87717d7524. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:42.428583 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:42.421297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.1.154.129:32873)
0620 19:07:42.421638 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:42.421648 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:42.421676 (+ 28us) server_negotiation.cc:407] Connection header received
0620 19:07:42.421759 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:42.421766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:42.422004 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:42.422470 (+ 466us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:42.422630 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.423768 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.425385 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.426198 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.427898 (+ 1700us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:42.428141 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:42.428199 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 19:07:42.428296 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":138,"threads_started":1}
I20260620 19:07:42.441293 14526 tablet_service.cc:1467] Processing CreateTablet for tablet d6d4660930d1469985241d87717d7524 (DEFAULT_TABLE table=test-table [id=a790eb8a10a64b409b7f9445ff49f3ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:42.442409 14526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6d4660930d1469985241d87717d7524. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:42.446038 14910 tablet_bootstrap.cc:492] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c: Bootstrap starting.
I20260620 19:07:42.449738 14910 tablet_bootstrap.cc:654] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:42.449826 14911 tablet_bootstrap.cc:492] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543: Bootstrap starting.
I20260620 19:07:42.450752 14910 log.cc:824] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:42.451538 14912 tablet_bootstrap.cc:492] T d6d4660930d1469985241d87717d7524 P 9283df07ab1148ffb4c378c504227818: Bootstrap starting.
I20260620 19:07:42.452229 14911 tablet_bootstrap.cc:654] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:42.453048 14911 log.cc:824] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:42.453452 14910 tablet_bootstrap.cc:492] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c: No bootstrap required, opened a new log
I20260620 19:07:42.453759 14910 ts_tablet_manager.cc:1397] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 19:07:42.457098 14912 tablet_bootstrap.cc:654] T d6d4660930d1469985241d87717d7524 P 9283df07ab1148ffb4c378c504227818: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:42.458128 14912 log.cc:824] T d6d4660930d1469985241d87717d7524 P 9283df07ab1148ffb4c378c504227818: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:42.458460 14910 raft_consensus.cc:348] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2076922b46d4fda948438069871351c" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33911 } } peers { permanent_uuid: "9283df07ab1148ffb4c378c504227818" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 32873 } } peers { permanent_uuid: "a8f0b9af2fbb49d1a03287ed375a3543" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 32867 } }
I20260620 19:07:42.458747 14910 raft_consensus.cc:374] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:42.458813 14910 raft_consensus.cc:729] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2076922b46d4fda948438069871351c, State: Initialized, Role: FOLLOWER
I20260620 19:07:42.458915 14911 tablet_bootstrap.cc:492] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543: No bootstrap required, opened a new log
I20260620 19:07:42.459118 14911 ts_tablet_manager.cc:1397] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20260620 19:07:42.459311 14910 consensus_queue.cc:260] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2076922b46d4fda948438069871351c" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33911 } } peers { permanent_uuid: "9283df07ab1148ffb4c378c504227818" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 32873 } } peers { permanent_uuid: "a8f0b9af2fbb49d1a03287ed375a3543" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 32867 } }
I20260620 19:07:42.460131 14884 heartbeater.cc:503] Master 127.1.154.190:33851 was elected leader, sending a full tablet report...
I20260620 19:07:42.460402 14910 ts_tablet_manager.cc:1428] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:07:42.466346 14911 raft_consensus.cc:348] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2076922b46d4fda948438069871351c" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33911 } } peers { permanent_uuid: "9283df07ab1148ffb4c378c504227818" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 32873 } } peers { permanent_uuid: "a8f0b9af2fbb49d1a03287ed375a3543" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 32867 } }
I20260620 19:07:42.466702 14911 raft_consensus.cc:374] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:42.466761 14911 raft_consensus.cc:729] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f0b9af2fbb49d1a03287ed375a3543, State: Initialized, Role: FOLLOWER
I20260620 19:07:42.467335 14911 consensus_queue.cc:260] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2076922b46d4fda948438069871351c" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33911 } } peers { permanent_uuid: "9283df07ab1148ffb4c378c504227818" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 32873 } } peers { permanent_uuid: "a8f0b9af2fbb49d1a03287ed375a3543" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 32867 } }
I20260620 19:07:42.468685 14911 ts_tablet_manager.cc:1428] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20260620 19:07:42.469100 14744 heartbeater.cc:503] Master 127.1.154.190:33851 was elected leader, sending a full tablet report...
I20260620 19:07:42.472947 14912 tablet_bootstrap.cc:492] T d6d4660930d1469985241d87717d7524 P 9283df07ab1148ffb4c378c504227818: No bootstrap required, opened a new log
I20260620 19:07:42.473197 14912 ts_tablet_manager.cc:1397] T d6d4660930d1469985241d87717d7524 P 9283df07ab1148ffb4c378c504227818: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20260620 19:07:42.477682 14912 raft_consensus.cc:348] T d6d4660930d1469985241d87717d7524 P 9283df07ab1148ffb4c378c504227818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2076922b46d4fda948438069871351c" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33911 } } peers { permanent_uuid: "9283df07ab1148ffb4c378c504227818" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 32873 } } peers { permanent_uuid: "a8f0b9af2fbb49d1a03287ed375a3543" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 32867 } }
I20260620 19:07:42.477998 14912 raft_consensus.cc:374] T d6d4660930d1469985241d87717d7524 P 9283df07ab1148ffb4c378c504227818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:42.478041 14912 raft_consensus.cc:729] T d6d4660930d1469985241d87717d7524 P 9283df07ab1148ffb4c378c504227818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9283df07ab1148ffb4c378c504227818, State: Initialized, Role: FOLLOWER
I20260620 19:07:42.478730 14912 consensus_queue.cc:260] T d6d4660930d1469985241d87717d7524 P 9283df07ab1148ffb4c378c504227818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2076922b46d4fda948438069871351c" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33911 } } peers { permanent_uuid: "9283df07ab1148ffb4c378c504227818" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 32873 } } peers { permanent_uuid: "a8f0b9af2fbb49d1a03287ed375a3543" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 32867 } }
I20260620 19:07:42.482075 14912 ts_tablet_manager.cc:1428] T d6d4660930d1469985241d87717d7524 P 9283df07ab1148ffb4c378c504227818: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
I20260620 19:07:42.483064 14609 heartbeater.cc:503] Master 127.1.154.190:33851 was elected leader, sending a full tablet report...
W20260620 19:07:42.483166 14610 tablet.cc:2369] T d6d4660930d1469985241d87717d7524 P 9283df07ab1148ffb4c378c504227818: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:07:42.490581 14916 raft_consensus.cc:482] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:42.490810 14916 raft_consensus.cc:504] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2076922b46d4fda948438069871351c" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33911 } } peers { permanent_uuid: "9283df07ab1148ffb4c378c504227818" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 32873 } } peers { permanent_uuid: "a8f0b9af2fbb49d1a03287ed375a3543" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 32867 } }
I20260620 19:07:42.492271 14916 leader_election.cc:288] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9283df07ab1148ffb4c378c504227818 (127.1.154.129:32873), a8f0b9af2fbb49d1a03287ed375a3543 (127.1.154.130:32867)
I20260620 19:07:42.499853 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:42.493281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:35887 (local address 127.1.154.129:32873)
0620 19:07:42.493465 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:42.493470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:42.495562 (+ 2092us) server_negotiation.cc:407] Connection header received
0620 19:07:42.495760 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:42.495763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:42.495812 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:42.495863 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:42.496252 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.496961 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.498461 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.499054 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.499158 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:42.499209 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:42.499646 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 19:07:42.499715 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 19:07:42.499853 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:42.493477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:39041 (local address 127.1.154.130:32867)
0620 19:07:42.493663 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:42.493668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:42.495212 (+ 1544us) server_negotiation.cc:407] Connection header received
0620 19:07:42.495376 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:42.495380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:42.495436 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:42.495511 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:42.495931 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.496730 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.498461 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.499054 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.499158 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:42.499209 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:42.499638 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 19:07:42.499715 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:07:42.500370 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:42.492701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:32873 (local address 127.1.154.131:35887)
0620 19:07:42.495487 (+ 2786us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:42.495501 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:42.495634 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:42.495998 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:42.496001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:42.496015 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:42.496128 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:42.496134 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.497078 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.497081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:42.498316 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:42.498327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.499309 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.499312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:42.499491 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:42.499528 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 19:07:42.500077 (+ 549us) client_negotiation.cc:240] Negotiation successful
0620 19:07:42.500187 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2712,"mutex_wait_us":134,"thread_start_us":1936,"threads_started":1}
I20260620 19:07:42.500370 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:42.492695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:32867 (local address 127.1.154.131:39041)
0620 19:07:42.495063 (+ 2368us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:42.495077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:42.495245 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:42.495638 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:42.495641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:42.495656 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:42.495810 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:42.495816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.496862 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.496865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:42.498319 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:42.498327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.499309 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.499312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:42.499465 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:42.499513 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 19:07:42.500077 (+ 564us) client_negotiation.cc:240] Negotiation successful
0620 19:07:42.500187 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2304,"mutex_wait_us":107,"thread_start_us":1922,"threads_started":1}
I20260620 19:07:42.501063 14698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6d4660930d1469985241d87717d7524" candidate_uuid: "b2076922b46d4fda948438069871351c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f0b9af2fbb49d1a03287ed375a3543" is_pre_election: true
I20260620 19:07:42.501063 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6d4660930d1469985241d87717d7524" candidate_uuid: "b2076922b46d4fda948438069871351c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9283df07ab1148ffb4c378c504227818" is_pre_election: true
I20260620 19:07:42.501406 14563 raft_consensus.cc:2454] T d6d4660930d1469985241d87717d7524 P 9283df07ab1148ffb4c378c504227818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2076922b46d4fda948438069871351c in term 0.
I20260620 19:07:42.501406 14698 raft_consensus.cc:2454] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2076922b46d4fda948438069871351c in term 0.
I20260620 19:07:42.501933 14767 leader_election.cc:302] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9283df07ab1148ffb4c378c504227818, a8f0b9af2fbb49d1a03287ed375a3543, b2076922b46d4fda948438069871351c; no voters:
I20260620 19:07:42.502198 14916 raft_consensus.cc:2781] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:07:42.502280 14916 raft_consensus.cc:482] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:07:42.502346 14916 raft_consensus.cc:3037] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:42.503707 14916 raft_consensus.cc:504] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2076922b46d4fda948438069871351c" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33911 } } peers { permanent_uuid: "9283df07ab1148ffb4c378c504227818" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 32873 } } peers { permanent_uuid: "a8f0b9af2fbb49d1a03287ed375a3543" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 32867 } }
I20260620 19:07:42.504426 14698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6d4660930d1469985241d87717d7524" candidate_uuid: "b2076922b46d4fda948438069871351c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f0b9af2fbb49d1a03287ed375a3543"
I20260620 19:07:42.504426 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6d4660930d1469985241d87717d7524" candidate_uuid: "b2076922b46d4fda948438069871351c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9283df07ab1148ffb4c378c504227818"
I20260620 19:07:42.504554 14698 raft_consensus.cc:3037] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:42.504554 14563 raft_consensus.cc:3037] T d6d4660930d1469985241d87717d7524 P 9283df07ab1148ffb4c378c504227818 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:42.505867 14563 raft_consensus.cc:2454] T d6d4660930d1469985241d87717d7524 P 9283df07ab1148ffb4c378c504227818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2076922b46d4fda948438069871351c in term 1.
I20260620 19:07:42.505867 14698 raft_consensus.cc:2454] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2076922b46d4fda948438069871351c in term 1.
I20260620 19:07:42.506222 14765 leader_election.cc:302] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9283df07ab1148ffb4c378c504227818, a8f0b9af2fbb49d1a03287ed375a3543, b2076922b46d4fda948438069871351c; no voters:
I20260620 19:07:42.506536 14916 leader_election.cc:288] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [CANDIDATE]: Term 1 election: Requested vote from peers 9283df07ab1148ffb4c378c504227818 (127.1.154.129:32873), a8f0b9af2fbb49d1a03287ed375a3543 (127.1.154.130:32867)
I20260620 19:07:42.506606 14921 raft_consensus.cc:2781] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:42.506738 14921 raft_consensus.cc:686] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [term 1 LEADER]: Becoming Leader. State: Replica: b2076922b46d4fda948438069871351c, State: Running, Role: LEADER
I20260620 19:07:42.507100 14921 consensus_queue.cc:237] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2076922b46d4fda948438069871351c" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33911 } } peers { permanent_uuid: "9283df07ab1148ffb4c378c504227818" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 32873 } } peers { permanent_uuid: "a8f0b9af2fbb49d1a03287ed375a3543" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 32867 } }
I20260620 19:07:42.511397 14411 catalog_manager.cc:5495] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c reported cstate change: term changed from 0 to 1, leader changed from <none> to b2076922b46d4fda948438069871351c (127.1.154.131). New cstate: current_term: 1 leader_uuid: "b2076922b46d4fda948438069871351c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2076922b46d4fda948438069871351c" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9283df07ab1148ffb4c378c504227818" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 32873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8f0b9af2fbb49d1a03287ed375a3543" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20260620 19:07:42.545900 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:42.539140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32802 (local address 127.1.154.131:33911)
0620 19:07:42.539384 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:42.539390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:42.539421 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:07:42.539573 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:42.539577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:42.539676 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:42.539855 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:07:42.540471 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.541516 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.543023 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:42.543356 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:42.543472 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:42.543955 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:07:42.545042 (+ 1087us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:07:42.545234 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:42.545554 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 19:07:42.545688 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 19:07:42.554388 14698 raft_consensus.cc:1261] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543 [term 1 FOLLOWER]: Refusing update from remote peer b2076922b46d4fda948438069871351c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:42.555485 14921 consensus_queue.cc:1035] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8f0b9af2fbb49d1a03287ed375a3543" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:42.556659 14563 raft_consensus.cc:1261] T d6d4660930d1469985241d87717d7524 P 9283df07ab1148ffb4c378c504227818 [term 1 FOLLOWER]: Refusing update from remote peer b2076922b46d4fda948438069871351c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:42.557380 14921 consensus_queue.cc:1035] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9283df07ab1148ffb4c378c504227818" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 32873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:42.566386 14924 mvcc.cc:205] Tried to move back new op lower bound from 7299000166613192704 to 7299000166432354304. Current Snapshot: MvccSnapshot[applied={T|T < 7299000166613192704}]
I20260620 19:07:42.573136 14926 mvcc.cc:205] Tried to move back new op lower bound from 7299000166613192704 to 7299000166432354304. Current Snapshot: MvccSnapshot[applied={T|T < 7299000166613192704}]
W20260620 19:07:42.574463 14745 tablet.cc:2369] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:07:42.578114 14925 mvcc.cc:205] Tried to move back new op lower bound from 7299000166613192704 to 7299000166432354304. Current Snapshot: MvccSnapshot[applied={T|T < 7299000166613192704}]
I20260620 19:07:42.589848 14411 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:42.590037 14411 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:42.593231 14411 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36d5ae91e0c14fc480fbd81fa79535c5: Sending DeleteTablet for 3 replicas of tablet d6d4660930d1469985241d87717d7524
I20260620 19:07:42.594192 14804 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d4660930d1469985241d87717d7524 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:42 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:32798
I20260620 19:07:42.595399 14675 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d4660930d1469985241d87717d7524 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:42 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:58798
I20260620 19:07:42.595786 14526 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d4660930d1469985241d87717d7524 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:42 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:57772
I20260620 19:07:42.596838 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 14449
I20260620 19:07:42.602972 14933 tablet_replica.cc:330] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c: stopping tablet replica
I20260620 19:07:42.603515 14933 raft_consensus.cc:2229] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:07:42.604182 14933 raft_consensus.cc:2258] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:42.604796 14934 tablet_replica.cc:330] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543: stopping tablet replica
I20260620 19:07:42.605207 14934 raft_consensus.cc:2229] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:42.605654 14934 raft_consensus.cc:2258] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:42.607173 14933 ts_tablet_manager.cc:1905] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:42.610217 14933 ts_tablet_manager.cc:1918] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:07:42.610332 14933 log.cc:1196] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/d6d4660930d1469985241d87717d7524
I20260620 19:07:42.610653 14933 ts_tablet_manager.cc:1939] T d6d4660930d1469985241d87717d7524 P b2076922b46d4fda948438069871351c: Deleting consensus metadata
I20260620 19:07:42.610852 14934 ts_tablet_manager.cc:1905] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:42.612023 14396 catalog_manager.cc:4841] TS b2076922b46d4fda948438069871351c (127.1.154.131:33911): tablet d6d4660930d1469985241d87717d7524 (table test-table [id=a790eb8a10a64b409b7f9445ff49f3ad]) successfully deleted
W20260620 19:07:42.622113 14398 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:32873 (error 108)
I20260620 19:07:42.625547 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:42.623955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:32873 (local address 127.0.0.1:57778)
0620 19:07:42.625275 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:42.625383 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1201}
I20260620 19:07:42.625684 14934 ts_tablet_manager.cc:1918] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:07:42.625891 14934 log.cc:1196] T d6d4660930d1469985241d87717d7524 P a8f0b9af2fbb49d1a03287ed375a3543: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/d6d4660930d1469985241d87717d7524
W20260620 19:07:42.625916 14398 catalog_manager.cc:4584] TS 9283df07ab1148ffb4c378c504227818 (127.1.154.129:32873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6d4660930d1469985241d87717d7524: Network error: Client connection negotiation failed: client connection to 127.1.154.129:32873: connect: Connection refused (error 111)
I20260620 19:07:42.626591 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 14613
I20260620 19:07:42.636643 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 14747
I20260620 19:07:42.639396 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:42.638964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:32867 (local address 127.0.0.1:58802)
0620 19:07:42.639210 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:42.639277 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.130:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260620 19:07:42.644248 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 14378
I20260620 19:07:42.652518 1642 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 19:07:42.999671 1642 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 19:07:42.999825 1642 mini_ranger.cc:66] Stopping Ranger...
I20260620 19:07:43.346939 1642 mini_ranger.cc:68] Stopped Ranger
2026-06-20 19:07:43.347 UTC [13445] LOG: received smart shutdown request
2026-06-20 19:07:43.351 UTC [13445] LOG: background worker "logical replication launcher" (PID 13454) exited with exit code 1
2026-06-20 19:07:43.351 UTC [13449] LOG: shutting down
2026-06-20 19:07:43.619 UTC [13445] LOG: database system is shut down
2026-06-20T19:07:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53444 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:07:43 dist-test-slave-6wrf krb5kdc[14945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:07:43 dist-test-slave-6wrf krb5kdc[14945](info): set up 2 sockets
Jun 20 19:07:43 dist-test-slave-6wrf krb5kdc[14945](info): commencing operation
krb5kdc: starting...
W20260620 19:07:44.874281 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:07:44 dist-test-slave-6wrf krb5kdc[14945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:07:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:07:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:07:45.013814 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43333
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:43877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:43333,127.1.154.189:36237,127.1.154.188:45833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:45.132500 14958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:45.132835 14958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:45.132921 14958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:45.137112 14958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:07:45.137214 14958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:45.137249 14958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:45.137316 14958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:07:45.137354 14958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:07:45.142913 14958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:43877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:43333,127.1.154.189:36237,127.1.154.188:45833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:45.144121 14958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:45.146014 14958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:45.157456 14963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:45.157456 14964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:45.157739 14966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:45.159785 14958 server_base.cc:1034] running on GCE node
I20260620 19:07:45.160596 14958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:45.161890 14958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:45.163362 14958 hybrid_clock.cc:648] HybridClock initialized: now 1781982465163296 us; error 74 us; skew 500 ppm
Jun 20 19:07:45 dist-test-slave-6wrf krb5kdc[14945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:45.168231 14958 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:07:45.169363 14958 webserver.cc:460] Webserver started at http://127.1.154.190:35487/ using document root <none> and password file <none>
I20260620 19:07:45.170039 14958 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:45.170137 14958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:45.170387 14958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:45.172962 14958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "3579f068e04445e88587834921989b7a"
format_stamp: "Formatted at 2026-06-20 19:07:45 on dist-test-slave-6wrf"
server_key: "92dbfe8b55826b1d8de5a38952815e35"
server_key_iv: "99f4865bb53a5968ae0839a728e6dc5d"
server_key_version: "encryptionkey@0"
I20260620 19:07:45.173700 14958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "3579f068e04445e88587834921989b7a"
format_stamp: "Formatted at 2026-06-20 19:07:45 on dist-test-slave-6wrf"
server_key: "92dbfe8b55826b1d8de5a38952815e35"
server_key_iv: "99f4865bb53a5968ae0839a728e6dc5d"
server_key_version: "encryptionkey@0"
I20260620 19:07:45.178578 14958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 19:07:45.182821 14973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:45.184952 14958 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:07:45.185372 14958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "3579f068e04445e88587834921989b7a"
format_stamp: "Formatted at 2026-06-20 19:07:45 on dist-test-slave-6wrf"
server_key: "92dbfe8b55826b1d8de5a38952815e35"
server_key_iv: "99f4865bb53a5968ae0839a728e6dc5d"
server_key_version: "encryptionkey@0"
I20260620 19:07:45.185585 14958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:45.203814 14958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:45.209080 14958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:45.209355 14958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:45.217908 14958 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:43333
I20260620 19:07:45.217980 15025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:43333 every 8 connection(s)
I20260620 19:07:45.219308 14958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:07:45.222703 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 14958
I20260620 19:07:45.222841 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:07:45.223125 1642 external_mini_cluster.cc:1428] Setting key b8f1d4a17fa84137a7cf89a378ab741f
I20260620 19:07:45.223592 15026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:45.227905 15026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.154.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.189@KRBTEST.COM" created.
Jun 20 19:07:45 dist-test-slave-6wrf krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:45.246887 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.230447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.1.154.190:43333)
0620 19:07:45.230939 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:45.230957 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:45.231221 (+ 264us) server_negotiation.cc:407] Connection header received
0620 19:07:45.232371 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:45.232376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:45.232584 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:45.232821 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:45.233805 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.234710 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.235788 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.236451 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.236909 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.240809 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:45.240882 (+ 73us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:45.240907 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:45.240943 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:45.243241 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.243841 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.243855 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.243857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.243973 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.244338 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.244345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.244347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.244968 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:45.245187 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:45.245691 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 19:07:45.245993 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":340,"thread_start_us":159,"threads_started":1}
I20260620 19:07:45.246887 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.230448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43333 (local address 127.0.0.1:52736)
0620 19:07:45.231064 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.231108 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:45.232057 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:45.232813 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:45.232819 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:45.233240 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:45.233572 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.233587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.234843 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.234846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.235645 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.235654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.236893 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.236898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.237114 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.238270 (+ 1156us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:45.238288 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:45.240363 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:45.243404 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.243416 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.243431 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.243713 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.244076 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.244080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.244082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.244251 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.245167 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:45.245174 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:45.245340 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 19:07:45.245550 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 19:07:45.246230 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":187,"spinlock_wait_cycles":13440,"thread_start_us":155,"threads_started":1}
I20260620 19:07:45.250875 15026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:07:45.252194 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.251839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:36237 (local address 127.0.0.1:33330)
0620 19:07:45.252032 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.252118 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:36237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 19:07:45.252502 14977 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.154.189:36237: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.189.keytab.
Entry for principal kudu/127.1.154.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.189.keytab.
I20260620 19:07:45.254367 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.254015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:36237 (local address 127.0.0.1:33336)
0620 19:07:45.254250 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.254303 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:36237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260620 19:07:45.254539 15026 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.189:36237: Network error: Client connection negotiation failed: client connection to 127.1.154.189:36237: connect: Connection refused (error 111)
I20260620 19:07:45.255071 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.189:36237
--webserver_interface=127.1.154.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.189.keytab
--principal=kudu/127.1.154.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:43877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:43333,127.1.154.189:36237,127.1.154.188:45833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:07:45.303857 15026 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } attempt: 1
I20260620 19:07:45.304984 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.304602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:36237 (local address 127.0.0.1:33346)
0620 19:07:45.304788 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.304861 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:36237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 19:07:45.306113 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.305668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:36237 (local address 127.0.0.1:33358)
0620 19:07:45.305906 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.306005 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:36237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260620 19:07:45.306406 15026 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.189:36237: Network error: Client connection negotiation failed: client connection to 127.1.154.189:36237: connect: Connection refused (error 111)
I20260620 19:07:45.374648 15026 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } attempt: 2
I20260620 19:07:45.375975 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.375491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:36237 (local address 127.0.0.1:33366)
0620 19:07:45.375792 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.375863 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:36237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260620 19:07:45.376852 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.376613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:36237 (local address 127.0.0.1:33368)
0620 19:07:45.376730 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.376786 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:36237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 19:07:45.377128 15026 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.189:36237: Network error: Client connection negotiation failed: client connection to 127.1.154.189:36237: connect: Connection refused (error 111)
W20260620 19:07:45.380756 15033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:45.381054 15033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:45.381100 15033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:45.385301 15033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:07:45.385377 15033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:45.385401 15033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:45.385421 15033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:07:45.385485 15033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:07:45.391366 15033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:43877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:43333,127.1.154.189:36237,127.1.154.188:45833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.189.keytab
--principal=kudu/127.1.154.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.189:36237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.154.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:45.392611 15033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:45.393761 15033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:45.402767 15038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:45.402809 15039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:45.402765 15041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:45.403159 15033 server_base.cc:1034] running on GCE node
I20260620 19:07:45.403698 15033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:45.404850 15033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:45.406073 15033 hybrid_clock.cc:648] HybridClock initialized: now 1781982465406027 us; error 60 us; skew 500 ppm
Jun 20 19:07:45 dist-test-slave-6wrf krb5kdc[14945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:45.409224 15033 init.cc:377] Logged in from keytab as kudu/127.1.154.189@KRBTEST.COM (short username kudu)
I20260620 19:07:45.410279 15033 webserver.cc:460] Webserver started at http://127.1.154.189:46797/ using document root <none> and password file <none>
I20260620 19:07:45.410813 15033 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:45.410876 15033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:45.411067 15033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:45.413070 15033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/data/instance:
uuid: "82453857a35d4e339567e2bf7e2f94bd"
format_stamp: "Formatted at 2026-06-20 19:07:45 on dist-test-slave-6wrf"
server_key: "dc4e547a5babd1c541e598eb1f492d69"
server_key_iv: "4f8be7c12fac1f95660a5c36b6707a00"
server_key_version: "encryptionkey@0"
I20260620 19:07:45.413589 15033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/wal/instance:
uuid: "82453857a35d4e339567e2bf7e2f94bd"
format_stamp: "Formatted at 2026-06-20 19:07:45 on dist-test-slave-6wrf"
server_key: "dc4e547a5babd1c541e598eb1f492d69"
server_key_iv: "4f8be7c12fac1f95660a5c36b6707a00"
server_key_version: "encryptionkey@0"
I20260620 19:07:45.418848 15033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 19:07:45.422695 15048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:45.424437 15033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260620 19:07:45.424574 15033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/wal
uuid: "82453857a35d4e339567e2bf7e2f94bd"
format_stamp: "Formatted at 2026-06-20 19:07:45 on dist-test-slave-6wrf"
server_key: "dc4e547a5babd1c541e598eb1f492d69"
server_key_iv: "4f8be7c12fac1f95660a5c36b6707a00"
server_key_version: "encryptionkey@0"
I20260620 19:07:45.424695 15033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:45.450614 15033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:45.454569 15033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:45.454754 15033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:45.462692 15033 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.189:36237
I20260620 19:07:45.462805 15100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.189:36237 every 8 connection(s)
I20260620 19:07:45.463757 15033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/data/info.pb
I20260620 19:07:45.466922 15101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:45.468343 15026 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } attempt: 3
I20260620 19:07:45.471634 15101 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:07:45.473645 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 15033
I20260620 19:07:45.473770 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-1/wal/instance
I20260620 19:07:45.474072 1642 external_mini_cluster.cc:1428] Setting key f6647e507181fbef6bcfb2c135630743
Jun 20 19:07:45 dist-test-slave-6wrf krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for kudu/127.1.154.189@KRBTEST.COM
Jun 20 19:07:45 dist-test-slave-6wrf krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.189@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.154.188@KRBTEST.COM; defaulting to no policy
I20260620 19:07:45.483770 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.468953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:36237 (local address 127.0.0.1:33378)
0620 19:07:45.469129 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.469148 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:45.469249 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:45.471943 (+ 2694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:45.471950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:45.472226 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:45.472356 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.472364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.474126 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.474130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.474911 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.474921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.475885 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.475890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.476169 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.476861 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:45.476877 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:45.478365 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:45.481661 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.481667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.481669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.481945 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.482368 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.482372 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.482374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.482477 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.483217 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:45.483220 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:45.483323 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 19:07:45.483449 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 19:07:45.483615 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":5}
I20260620 19:07:45.485373 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.469689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.1.154.189:36237)
0620 19:07:45.470132 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:45.470151 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:45.470195 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:07:45.471036 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:45.471069 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:45.471492 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:45.471856 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:45.472518 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.473985 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.475036 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.475518 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.476009 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.478555 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:45.478561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:45.478570 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:45.478596 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:45.481503 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.482074 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.482082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.482090 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.482255 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.482592 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.482595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.482597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.483095 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:45.483238 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:45.483981 (+ 743us) server_negotiation.cc:299] Negotiation successful
0620 19:07:45.484314 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.188@KRBTEST.COM" created.
I20260620 19:07:45.486860 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.473680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.1.154.190:43333)
0620 19:07:45.473805 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:45.473811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:45.474496 (+ 685us) server_negotiation.cc:407] Connection header received
0620 19:07:45.474703 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:45.474706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:45.474753 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:45.474836 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:45.475862 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.477437 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.479437 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.480100 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.480226 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.482954 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:45.482959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:45.482962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:45.482990 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:45.484852 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.485349 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.485355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.485358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.485448 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.485856 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.485859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.485861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.486105 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:45.486201 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:45.486647 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 19:07:45.486762 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:07:45.487279 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.473681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43333 (local address 127.0.0.1:52750)
0620 19:07:45.474356 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.474408 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:45.474592 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:45.474923 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:45.474931 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:45.475470 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:45.475700 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.475711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.477547 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.477553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.478712 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.478726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.480226 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.480230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.480459 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.480990 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:45.481004 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:45.482798 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:45.484986 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.484993 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.484997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.485226 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.485637 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.485640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.485642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.485735 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.486277 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:45.486282 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:45.486392 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 19:07:45.486997 (+ 605us) client_negotiation.cc:240] Negotiation successful
0620 19:07:45.487154 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":5,"thread_start_us":208,"threads_started":1}
I20260620 19:07:45.488266 15026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:07:45.488567 15101 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:07:45.489144 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.488782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48156)
0620 19:07:45.489016 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.489079 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260620 19:07:45.489961 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.489703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48158)
0620 19:07:45.489850 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.489898 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 19:07:45.490163 15026 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:45833: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Jun 20 19:07:45 dist-test-slave-6wrf krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.189@KRBTEST.COM for kudu/127.1.154.189@KRBTEST.COM
I20260620 19:07:45.500496 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.489048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:36237 (local address 127.0.0.1:33384)
0620 19:07:45.489183 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.489195 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:45.489274 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:45.489559 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:45.489564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:45.489791 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:45.489932 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.489942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.491508 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.491512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.492082 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.492091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.492816 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.492819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.492895 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.493449 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:45.493460 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:45.495647 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:45.498256 (+ 2609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.498260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.498262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.498658 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.499031 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.499034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.499037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.499136 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.499992 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:45.499995 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:45.500111 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 19:07:45.500222 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 19:07:45.500355 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 19:07:45.500747 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.489071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33384 (local address 127.1.154.189:36237)
0620 19:07:45.489165 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:45.489169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:45.489372 (+ 203us) server_negotiation.cc:407] Connection header received
0620 19:07:45.489401 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:45.489403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:45.489441 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:45.489507 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:45.490055 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.491364 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.492200 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.492675 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.492807 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.495831 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:45.495834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:45.495836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:45.495859 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:45.498088 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.498828 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.498831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.498832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.498912 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.499476 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.499479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.499481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.499820 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:45.499936 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:45.500301 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 19:07:45.500439 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":30}
I20260620 19:07:45.501833 15101 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:07:45.502867 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.502527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48162)
0620 19:07:45.502669 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.502790 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 19:07:45.503087 15049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
I20260620 19:07:45.503922 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.503680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48174)
0620 19:07:45.503786 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.503837 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 19:07:45.504119 15101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:45833: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.188.keytab.
Entry for principal kudu/127.1.154.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.188.keytab.
I20260620 19:07:45.505728 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.188:45833
--webserver_interface=127.1.154.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.188.keytab
--principal=kudu/127.1.154.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:43877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:43333,127.1.154.189:36237,127.1.154.188:45833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:07:45.521391 15026 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } attempt: 1
I20260620 19:07:45.522316 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.522028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48180)
0620 19:07:45.522176 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.522243 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 19:07:45.523046 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.522836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48186)
0620 19:07:45.522944 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.522991 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 19:07:45.523193 15026 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:45833: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
I20260620 19:07:45.553491 15101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } attempt: 1
I20260620 19:07:45.554785 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.554432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48200)
0620 19:07:45.554630 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.554697 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 19:07:45.555658 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.555415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48204)
0620 19:07:45.555559 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.555606 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 19:07:45.555860 15101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:45833: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
I20260620 19:07:45.598407 15026 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } attempt: 2
I20260620 19:07:45.599774 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.599391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48218)
0620 19:07:45.599586 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.599671 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 19:07:45.600869 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.600602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48228)
0620 19:07:45.600727 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.600790 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 19:07:45.601358 15026 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:45833: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
I20260620 19:07:45.624064 15101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } attempt: 2
I20260620 19:07:45.625034 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.624697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48244)
0620 19:07:45.624893 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.624961 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 19:07:45.626271 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.625880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48246)
0620 19:07:45.626162 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.626219 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 19:07:45.626443 15101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:45833: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
W20260620 19:07:45.628815 15108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:45.629086 15108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:45.629187 15108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:45.633669 15108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:07:45.633733 15108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:45.633752 15108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:45.633774 15108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:07:45.633797 15108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:07:45.639441 15108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:43877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:43333,127.1.154.189:36237,127.1.154.188:45833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.188.keytab
--principal=kudu/127.1.154.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.188:45833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.154.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:45.640575 15108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:45.641700 15108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:45.652604 15113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:45.652878 15108 server_base.cc:1034] running on GCE node
W20260620 19:07:45.652684 15114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:45.652604 15116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:45.654202 15108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:45.655499 15108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:45.656735 15108 hybrid_clock.cc:648] HybridClock initialized: now 1781982465656676 us; error 80 us; skew 500 ppm
Jun 20 19:07:45 dist-test-slave-6wrf krb5kdc[14945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:45.660799 15108 init.cc:377] Logged in from keytab as kudu/127.1.154.188@KRBTEST.COM (short username kudu)
I20260620 19:07:45.661940 15108 webserver.cc:460] Webserver started at http://127.1.154.188:34873/ using document root <none> and password file <none>
I20260620 19:07:45.662528 15108 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:45.662618 15108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:45.662855 15108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:45.664947 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/data/instance:
uuid: "31c4df8a43404d59a96444ab3b538963"
format_stamp: "Formatted at 2026-06-20 19:07:45 on dist-test-slave-6wrf"
server_key: "b4406466cd5b783e0b71518426d73ad0"
server_key_iv: "1cb9ebe0ae491e1371c943691638a109"
server_key_version: "encryptionkey@0"
I20260620 19:07:45.665515 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/wal/instance:
uuid: "31c4df8a43404d59a96444ab3b538963"
format_stamp: "Formatted at 2026-06-20 19:07:45 on dist-test-slave-6wrf"
server_key: "b4406466cd5b783e0b71518426d73ad0"
server_key_iv: "1cb9ebe0ae491e1371c943691638a109"
server_key_version: "encryptionkey@0"
I20260620 19:07:45.670341 15108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:07:45.674150 15123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:45.675762 15108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:07:45.675901 15108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/wal
uuid: "31c4df8a43404d59a96444ab3b538963"
format_stamp: "Formatted at 2026-06-20 19:07:45 on dist-test-slave-6wrf"
server_key: "b4406466cd5b783e0b71518426d73ad0"
server_key_iv: "1cb9ebe0ae491e1371c943691638a109"
server_key_version: "encryptionkey@0"
I20260620 19:07:45.676083 15108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:45.701808 15026 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } attempt: 3
I20260620 19:07:45.702961 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.702613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48252)
0620 19:07:45.702767 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.702838 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 19:07:45.703886 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.703590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48254)
0620 19:07:45.703757 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.703805 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260620 19:07:45.704078 15026 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:45833: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
I20260620 19:07:45.715389 15108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:45.717741 15101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } attempt: 3
I20260620 19:07:45.718961 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.718651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48262)
0620 19:07:45.718812 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.718874 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 19:07:45.720444 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.719906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48278)
0620 19:07:45.720338 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.720387 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260620 19:07:45.720638 15101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:45833: Network error: Client connection negotiation failed: client connection to 127.1.154.188:45833: connect: Connection refused (error 111)
I20260620 19:07:45.722113 15108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:45.722354 15108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:45.734074 15108 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.188:45833
I20260620 19:07:45.734438 15175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.188:45833 every 8 connection(s)
I20260620 19:07:45.735562 15108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/data/info.pb
I20260620 19:07:45.738932 15176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:45.743286 15176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:07:45.744678 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 15108
I20260620 19:07:45.744822 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982293404112-1642-0/minicluster-data/master-2/wal/instance
I20260620 19:07:45.745098 1642 external_mini_cluster.cc:1428] Setting key 9e6a4e4ce7715214215b7bae0cfd10fa
Jun 20 19:07:45 dist-test-slave-6wrf krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.188@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
Jun 20 19:07:45 dist-test-slave-6wrf krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.188@KRBTEST.COM
I20260620 19:07:45.765107 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.747619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52752 (local address 127.1.154.190:43333)
0620 19:07:45.747978 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:45.747984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:45.748566 (+ 582us) server_negotiation.cc:407] Connection header received
0620 19:07:45.749359 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:45.749364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:45.749475 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:45.749634 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:45.752285 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.752925 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.754073 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.754319 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.754391 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.758400 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:45.758405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:45.758408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:45.758459 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:45.761663 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.762332 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.762338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.762343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.762452 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.762941 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.762944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.762946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.763190 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:45.763300 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:45.764297 (+ 997us) server_negotiation.cc:299] Negotiation successful
0620 19:07:45.764469 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 19:07:45.765573 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.746418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48282 (local address 127.1.154.188:45833)
0620 19:07:45.746898 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:45.746923 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:45.747130 (+ 207us) server_negotiation.cc:407] Connection header received
0620 19:07:45.748856 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:45.748891 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:45.749446 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:45.749687 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:45.751419 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.752586 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.754076 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.754867 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.755024 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.759568 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:45.759576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:45.759589 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:45.759615 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:45.762441 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.762914 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.762922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.762924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.763013 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.763319 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.763323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.763324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.763718 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:45.763947 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:45.764198 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 19:07:45.764634 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":311,"thread_start_us":151,"threads_started":1}
I20260620 19:07:45.766042 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.746420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43333 (local address 127.0.0.1:52752)
0620 19:07:45.748263 (+ 1843us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.748353 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:45.748876 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:45.749937 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:45.749946 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:45.751461 (+ 1515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:45.752067 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.752088 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.753048 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.753052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.753931 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.753944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.754566 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.754568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.754709 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.755755 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:45.755785 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:45.758087 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:45.761803 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.761811 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.761824 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.762195 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.762575 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.762581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.762585 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.762805 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.763378 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:45.763388 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:45.763784 (+ 396us) client_negotiation.cc:769] Sending connection context
0620 19:07:45.765354 (+ 1570us) client_negotiation.cc:240] Negotiation successful
0620 19:07:45.765617 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1550,"mutex_wait_us":34,"thread_start_us":269,"threads_started":1}
I20260620 19:07:45.768543 15176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:07:45 dist-test-slave-6wrf krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.188@KRBTEST.COM for kudu/127.1.154.189@KRBTEST.COM
Jun 20 19:07:45 dist-test-slave-6wrf krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:45.782693 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.769180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:36237 (local address 127.0.0.1:33386)
0620 19:07:45.769363 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.769377 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:45.769477 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:45.769900 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:45.769902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:45.770123 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:45.770249 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.770254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.771609 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.771612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.772279 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.772288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.772764 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.772768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.772864 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.773566 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:45.773580 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:45.775330 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:45.779965 (+ 4635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.779971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.779975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.780497 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.780986 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.780990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.780992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.781111 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.781964 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:45.781967 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:45.782086 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 19:07:45.782205 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 19:07:45.782366 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 19:07:45.783299 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.769271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.1.154.189:36237)
0620 19:07:45.769386 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:45.769393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:45.769426 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:07:45.769623 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:45.769626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:45.769682 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:45.769739 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:45.770709 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.771440 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.772424 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.772650 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.772764 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.775963 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:45.775971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:45.775974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:45.776030 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:45.779787 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.780654 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.780659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.780662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.780784 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.781266 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.781271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.781276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.781629 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:45.782022 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:45.782593 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 19:07:45.782825 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:07:45.784087 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.770024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.1.154.190:43333)
0620 19:07:45.770192 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:45.770198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:45.770217 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:07:45.770277 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:45.770282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:45.770341 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:45.770452 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:45.771542 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.772574 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.774047 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.774329 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.774450 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.778347 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:45.778355 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:45.778364 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:45.778408 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:45.781956 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.782542 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.782546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.782548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.782638 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.782970 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.782975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.782978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.783178 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:45.783403 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:45.783782 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 19:07:45.783913 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:07:45.784178 15176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:07:45 dist-test-slave-6wrf krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.189@KRBTEST.COM
Jun 20 19:07:45 dist-test-slave-6wrf krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.188@KRBTEST.COM for kudu/127.1.154.188@KRBTEST.COM
I20260620 19:07:45.797732 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.785707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.1.154.189:36237)
0620 19:07:45.785823 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:45.785827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:45.785841 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:07:45.785933 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:45.785936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:45.785975 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:45.786070 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:45.786792 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.787489 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.788282 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.788614 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.788810 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.792718 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:45.792724 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:45.792729 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:45.792766 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:45.795231 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.796014 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.796021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.796026 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.796134 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.796562 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.796567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.796571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.796856 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:45.797142 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:45.797250 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 19:07:45.797402 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:07:45.798007 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.785338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.1.154.188:45833)
0620 19:07:45.785454 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:45.785461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:45.785475 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:07:45.785514 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:45.785518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:45.785570 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:45.785680 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:45.786242 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.787056 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.788903 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.790657 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.790807 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.793680 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:45.793686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:45.793688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:45.793717 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:45.796066 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.796487 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.796492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.796494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.796568 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.796840 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.796842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.796844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.797070 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:45.797159 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:45.797656 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 19:07:45.797821 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":38}
I20260620 19:07:45.798038 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.785098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48298)
0620 19:07:45.785220 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.785235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:45.785317 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:45.785800 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:45.785805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:45.786015 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:45.786128 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.786137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.787209 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.787211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.788630 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.788661 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.790865 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.790873 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.791101 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.791767 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:45.791787 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:45.793482 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:45.796177 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.796183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.796185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.796386 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.796662 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.796665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.796667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.796748 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.797202 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:45.797205 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:45.797282 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 19:07:45.797381 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 19:07:45.797905 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":91}
I20260620 19:07:45.801649 15176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963: Bootstrap starting.
I20260620 19:07:45.802575 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:07:45.804627 15176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:45.805545 15176 log.cc:824] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:45.817070 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.805744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.1.154.189:36237)
0620 19:07:45.806029 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:45.806036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:45.806604 (+ 568us) server_negotiation.cc:407] Connection header received
0620 19:07:45.806847 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:45.806850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:45.806911 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:45.807002 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:45.807771 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.808551 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.809754 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.810098 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.810192 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.811947 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:45.811951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:45.811954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:45.811980 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:45.813795 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.814542 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.814551 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.814556 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.814833 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.815368 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.815373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.815375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.815694 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:45.816577 (+ 883us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:45.816686 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 19:07:45.816835 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260620 19:07:45.820165 15176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963: No bootstrap required, opened a new log
I20260620 19:07:45.820488 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.806599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.1.154.188:45833)
0620 19:07:45.806741 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:45.806747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:45.807675 (+ 928us) server_negotiation.cc:407] Connection header received
0620 19:07:45.807882 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:45.807888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:45.807959 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:45.808024 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:45.809384 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.810617 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.812065 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.812528 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.812617 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.815034 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:45.815041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:45.815046 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:45.815081 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:45.817011 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.818068 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.818072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.818076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.818161 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.818764 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.818768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.818772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.819096 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:45.819218 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:45.819817 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 19:07:45.819992 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:07:45.823601 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.807156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.1.154.190:43333)
0620 19:07:45.807291 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:45.807296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:45.807971 (+ 675us) server_negotiation.cc:407] Connection header received
0620 19:07:45.808139 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:45.808142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:45.808195 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:45.808244 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:45.809208 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.810012 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.811458 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.811813 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.811891 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.814669 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:45.814673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:45.814677 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:45.814716 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:45.817193 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.817803 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.817807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.817809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.817900 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.822043 (+ 4143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.822049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.822052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.822355 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:45.822577 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:45.823170 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 19:07:45.823385 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:07:45.826772 15176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } }
I20260620 19:07:45.827013 15176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:45.827046 15176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c4df8a43404d59a96444ab3b538963, State: Initialized, Role: FOLLOWER
I20260620 19:07:45.827492 15176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } }
I20260620 19:07:45.828639 15189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } } }
I20260620 19:07:45.828791 15189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:07:45.829668 15176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 19:07:45.835469 15200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:07:45.835556 15200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:07:45.833317 15108 master_runner.cc:186] Error getting master registration for 127.1.154.190:43333: OK, instance_id { permanent_uuid: "3579f068e04445e88587834921989b7a" instance_seqno: 1781982465215457 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:07:45.836977 15108 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:07:45.863893 15101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } attempt: 4
I20260620 19:07:45.868353 15026 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } attempt: 4
Jun 20 19:07:45 dist-test-slave-6wrf krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.189@KRBTEST.COM for kudu/127.1.154.188@KRBTEST.COM
I20260620 19:07:45.875597 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.864633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48314)
0620 19:07:45.864903 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.864923 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:45.865054 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:45.865395 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:45.865398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:45.865717 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:45.865942 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.865957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.866892 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.866898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.867513 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.867524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.868017 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.868020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.868316 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.869203 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:45.869222 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:45.871328 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:45.873336 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.873339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.873341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.873585 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.873928 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.873931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.873933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.874020 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.875066 (+ 1046us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:45.875069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:45.875171 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 19:07:45.875257 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:07:45.875394 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260620 19:07:45.876353 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.864692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.1.154.188:45833)
0620 19:07:45.864919 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:45.864926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:45.864978 (+ 52us) server_negotiation.cc:407] Connection header received
0620 19:07:45.865224 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:45.865227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:45.865283 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:45.865378 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:45.866145 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.866758 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.867655 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.867891 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.868014 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.871486 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:45.871491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:45.871493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:45.871520 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:45.873215 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.873725 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.873729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.873731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.873810 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.874127 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.874130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.874131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.874357 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:45.874444 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:45.876015 (+ 1571us) server_negotiation.cc:299] Negotiation successful
0620 19:07:45.876147 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 19:07:45 dist-test-slave-6wrf krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for kudu/127.1.154.188@KRBTEST.COM
I20260620 19:07:45.880403 15101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd: Bootstrap starting.
I20260620 19:07:45.881969 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.869553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:45833 (local address 127.0.0.1:48316)
0620 19:07:45.869712 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:45.869731 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:45.869833 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:45.870533 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:45.870536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:45.870727 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:45.870853 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.870859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.874631 (+ 3772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.874634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.875411 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:45.875420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.876011 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.876015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:45.876097 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.876595 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:45.876615 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:45.878224 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:45.880353 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.880356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.880358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.880583 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.880887 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:45.880890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:45.880892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:45.880978 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:45.881444 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:45.881447 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:45.881542 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 19:07:45.881653 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 19:07:45.881786 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 19:07:45.882684 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:45.869870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48316 (local address 127.1.154.188:45833)
0620 19:07:45.870239 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:45.870252 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:45.870267 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:07:45.870312 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:45.870315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:45.870387 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:45.870967 (+ 580us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:45.871108 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.874516 (+ 3408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.875544 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:45.875883 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:45.878364 (+ 2481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:45.878465 (+ 101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:45.878469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:45.878471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:45.878486 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:45.880210 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.880704 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.880707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.880708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.880786 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:45.881088 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:45.881091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:45.881093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:45.881288 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:45.882260 (+ 972us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:45.882357 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 19:07:45.882522 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":134,"threads_started":1}
I20260620 19:07:45.884071 15101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:45.885877 15026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a: Bootstrap starting.
I20260620 19:07:45.886787 15101 log.cc:824] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:45.888253 15026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:45.888916 15026 log.cc:824] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:45.891101 15026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a: No bootstrap required, opened a new log
I20260620 19:07:45.894870 15026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } }
I20260620 19:07:45.895088 15026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:45.895144 15026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3579f068e04445e88587834921989b7a, State: Initialized, Role: FOLLOWER
I20260620 19:07:45.895334 15101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd: No bootstrap required, opened a new log
I20260620 19:07:45.895637 15026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } }
I20260620 19:07:45.896932 15026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:07:45.900853 15101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } }
I20260620 19:07:45.901162 15101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:45.901268 15101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82453857a35d4e339567e2bf7e2f94bd, State: Initialized, Role: FOLLOWER
I20260620 19:07:45.901794 15101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } }
I20260620 19:07:45.903192 15101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [sys.catalog]: configured and running, proceeding with master startup.
W20260620 19:07:45.908365 15228 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:07:45.908483 15228 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:07:45.903820 14958 master_runner.cc:186] Error getting master registration for 127.1.154.189:36237: OK, instance_id { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" instance_seqno: 1781982465460312 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:07:45.909479 14958 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:07:45.909775 15205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } } }
I20260620 19:07:45.909891 15205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:07:45.913764 15230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } } }
I20260620 19:07:45.913908 15230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:07:45.914005 15230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:45.914131 15230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } }
I20260620 19:07:45.915613 15000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82453857a35d4e339567e2bf7e2f94bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3579f068e04445e88587834921989b7a" is_pre_election: true
I20260620 19:07:45.915804 15033 master_runner.cc:263] No leader master found from master 82453857a35d4e339567e2bf7e2f94bd
I20260620 19:07:45.915863 15033 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:07:45.915927 15000 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82453857a35d4e339567e2bf7e2f94bd in term 0.
I20260620 19:07:45.916356 15050 leader_election.cc:302] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3579f068e04445e88587834921989b7a, 82453857a35d4e339567e2bf7e2f94bd; no voters:
I20260620 19:07:45.917013 15230 leader_election.cc:288] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3579f068e04445e88587834921989b7a (127.1.154.190:43333), 31c4df8a43404d59a96444ab3b538963 (127.1.154.188:45833)
I20260620 19:07:45.917158 15230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:07:45.917238 15230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:07:45.917296 15230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:45.917421 15150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82453857a35d4e339567e2bf7e2f94bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c4df8a43404d59a96444ab3b538963" is_pre_election: true
I20260620 19:07:45.917740 15150 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82453857a35d4e339567e2bf7e2f94bd in term 0.
I20260620 19:07:45.918970 15230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } }
I20260620 19:07:45.919642 15000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82453857a35d4e339567e2bf7e2f94bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3579f068e04445e88587834921989b7a"
I20260620 19:07:45.919723 15150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82453857a35d4e339567e2bf7e2f94bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c4df8a43404d59a96444ab3b538963"
I20260620 19:07:45.919760 15000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:45.919850 15150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:45.921140 15000 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82453857a35d4e339567e2bf7e2f94bd in term 1.
I20260620 19:07:45.921357 15150 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82453857a35d4e339567e2bf7e2f94bd in term 1.
I20260620 19:07:45.921571 15050 leader_election.cc:302] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3579f068e04445e88587834921989b7a, 82453857a35d4e339567e2bf7e2f94bd; no voters:
I20260620 19:07:45.921784 15230 leader_election.cc:288] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [CANDIDATE]: Term 1 election: Requested vote from peers 3579f068e04445e88587834921989b7a (127.1.154.190:43333), 31c4df8a43404d59a96444ab3b538963 (127.1.154.188:45833)
I20260620 19:07:45.921968 15230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:45.922065 15230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [term 1 LEADER]: Becoming Leader. State: Replica: 82453857a35d4e339567e2bf7e2f94bd, State: Running, Role: LEADER
I20260620 19:07:45.922464 15230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } }
W20260620 19:07:45.933921 15216 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 19:07:45.934250 15217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82453857a35d4e339567e2bf7e2f94bd. Latest consensus state: current_term: 1 leader_uuid: "82453857a35d4e339567e2bf7e2f94bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } } }
I20260620 19:07:45.934504 15217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [sys.catalog]: This master's current role is: LEADER
W20260620 19:07:45.934762 15216 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:07:45.938897 15233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:07:45.939944 15233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:07:45.947585 15000 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a [term 1 FOLLOWER]: Refusing update from remote peer 82453857a35d4e339567e2bf7e2f94bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:45.949378 15150 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963 [term 1 FOLLOWER]: Refusing update from remote peer 82453857a35d4e339567e2bf7e2f94bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:45.950084 15217 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:45.950562 15217 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:07:45.956434 15054 tablet.cc:2369] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:07:45.956332 15189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82453857a35d4e339567e2bf7e2f94bd. Latest consensus state: current_term: 1 leader_uuid: "82453857a35d4e339567e2bf7e2f94bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } } }
I20260620 19:07:45.956641 15189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:07:45.960448 15205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82453857a35d4e339567e2bf7e2f94bd. Latest consensus state: current_term: 1 leader_uuid: "82453857a35d4e339567e2bf7e2f94bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } } }
I20260620 19:07:45.960598 15205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:07:45.961678 14979 tablet.cc:2369] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:07:45.968012 15230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82453857a35d4e339567e2bf7e2f94bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } } }
I20260620 19:07:45.968158 15230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:45.968271 15234 mvcc.cc:205] Tried to move back new op lower bound from 7299000180507885568 to 7299000180454395904. Current Snapshot: MvccSnapshot[applied={T|T < 7299000180507885568}]
I20260620 19:07:45.969533 15205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82453857a35d4e339567e2bf7e2f94bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } } }
I20260620 19:07:45.969677 15205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:07:45.971274 15237 mvcc.cc:205] Tried to move back new op lower bound from 7299000180507885568 to 7299000180454395904. Current Snapshot: MvccSnapshot[applied={T|T < 7299000180507885568}]
I20260620 19:07:45.973412 15217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82453857a35d4e339567e2bf7e2f94bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } } }
I20260620 19:07:45.973529 15217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd [sys.catalog]: This master's current role is: LEADER
W20260620 19:07:45.974232 15129 tablet.cc:2369] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:07:45.974520 15189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82453857a35d4e339567e2bf7e2f94bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3579f068e04445e88587834921989b7a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43333 } } peers { permanent_uuid: "82453857a35d4e339567e2bf7e2f94bd" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 36237 } } peers { permanent_uuid: "31c4df8a43404d59a96444ab3b538963" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 45833 } } }
I20260620 19:07:45.974617 15189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:07:45.975976 15233 catalog_manager.cc:1347] Generated new cluster ID: fa26e8f7cc924423a440ee96a2a5a59c
I20260620 19:07:45.976049 15233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:07:45.977106 15235 mvcc.cc:205] Tried to move back new op lower bound from 7299000180507885568 to 7299000180454395904. Current Snapshot: MvccSnapshot[applied={T|T < 7299000180507885568}]
I20260620 19:07:46.012687 15233 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:07:46.014111 15233 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:07:46.035243 15233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82453857a35d4e339567e2bf7e2f94bd: Generated new TSK 0
I20260620 19:07:46.036201 15233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:07:46.836931 15200 catalog_manager.cc:1259] Loaded cluster ID: fa26e8f7cc924423a440ee96a2a5a59c
I20260620 19:07:46.837023 15200 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963: loading cluster ID for follower catalog manager: success
I20260620 19:07:46.838636 15200 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963: acquiring CA information for follower catalog manager: success
I20260620 19:07:46.839756 15200 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31c4df8a43404d59a96444ab3b538963: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:07:46.936723 15216 catalog_manager.cc:1259] Loaded cluster ID: fa26e8f7cc924423a440ee96a2a5a59c
I20260620 19:07:46.936856 15216 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a: loading cluster ID for follower catalog manager: success
I20260620 19:07:46.939186 15216 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a: acquiring CA information for follower catalog manager: success
I20260620 19:07:46.940968 15216 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3579f068e04445e88587834921989b7a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:07:47.117084 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 14958
I20260620 19:07:47.125288 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 15033
I20260620 19:07:47.132571 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 15108
2026-06-20T19:07:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3384 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:07:47 dist-test-slave-6wrf krb5kdc[15249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:07:47 dist-test-slave-6wrf krb5kdc[15249](info): set up 2 sockets
Jun 20 19:07:47 dist-test-slave-6wrf krb5kdc[15249](info): commencing operation
krb5kdc: starting...
W20260620 19:07:48.222051 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:07:48 dist-test-slave-6wrf krb5kdc[15249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:07:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:07:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:07:48.354693 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:37253
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:46259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:37253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:48.477406 15262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:48.478080 15262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:48.478191 15262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:48.483551 15262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:07:48.483657 15262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:48.483697 15262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:48.483736 15262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:07:48.483777 15262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:07:48.491070 15262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:46259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:37253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:37253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:48.492684 15262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:48.493832 15262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:48.502415 15270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:48.502592 15268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:48.502622 15267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:48.503113 15262 server_base.cc:1034] running on GCE node
I20260620 19:07:48.503697 15262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:48.504792 15262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:48.506026 15262 hybrid_clock.cc:648] HybridClock initialized: now 1781982468506019 us; error 43 us; skew 500 ppm
Jun 20 19:07:48 dist-test-slave-6wrf krb5kdc[15249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982468, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:48.508908 15262 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:07:48.509907 15262 webserver.cc:460] Webserver started at http://127.1.154.190:45623/ using document root <none> and password file <none>
I20260620 19:07:48.510475 15262 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:48.510569 15262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:48.510800 15262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:48.512878 15262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "dfafd442c86d474189fbe26e05e97347"
format_stamp: "Formatted at 2026-06-20 19:07:48 on dist-test-slave-6wrf"
server_key: "0526420eb35d31ed3bd82b269e0d46ae"
server_key_iv: "2707113feb0bb8480bd95fd7b94a434b"
server_key_version: "encryptionkey@0"
I20260620 19:07:48.513526 15262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "dfafd442c86d474189fbe26e05e97347"
format_stamp: "Formatted at 2026-06-20 19:07:48 on dist-test-slave-6wrf"
server_key: "0526420eb35d31ed3bd82b269e0d46ae"
server_key_iv: "2707113feb0bb8480bd95fd7b94a434b"
server_key_version: "encryptionkey@0"
I20260620 19:07:48.518214 15262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:07:48.520900 15277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:48.522434 15262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:07:48.522588 15262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "dfafd442c86d474189fbe26e05e97347"
format_stamp: "Formatted at 2026-06-20 19:07:48 on dist-test-slave-6wrf"
server_key: "0526420eb35d31ed3bd82b269e0d46ae"
server_key_iv: "2707113feb0bb8480bd95fd7b94a434b"
server_key_version: "encryptionkey@0"
I20260620 19:07:48.522707 15262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:48.533542 15262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:48.538173 15262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:48.538472 15262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:48.549466 15262 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:37253
I20260620 19:07:48.549504 15329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:37253 every 8 connection(s)
I20260620 19:07:48.550580 15262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:07:48.553000 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 15262
I20260620 19:07:48.553151 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:07:48.553475 1642 external_mini_cluster.cc:1428] Setting key 2f0c682499771bc711f2010cb4276c84
I20260620 19:07:48.554255 15330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:48.560191 15330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347: Bootstrap starting.
Jun 20 19:07:48 dist-test-slave-6wrf krb5kdc[15249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:48.562976 15330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:48.563891 15330 log.cc:824] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:48.566676 15330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347: No bootstrap required, opened a new log
I20260620 19:07:48.569713 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:48.554935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50016 (local address 127.1.154.190:37253)
0620 19:07:48.555368 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:48.555384 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:48.555420 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:07:48.556059 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:48.556074 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:48.556400 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:48.556755 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:48.557569 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:48.558621 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:48.559471 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:48.559935 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:48.560109 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:48.563581 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:48.563588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:48.563606 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:48.563644 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:48.566072 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:48.566619 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:48.566625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:48.566632 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:48.566778 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:48.567141 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:48.567144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:48.567145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:48.567670 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:48.567873 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:48.568180 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 19:07:48.568515 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":144,"threads_started":1}
I20260620 19:07:48.571532 15330 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfafd442c86d474189fbe26e05e97347" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37253 } }
I20260620 19:07:48.571811 15330 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:48.571898 15330 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfafd442c86d474189fbe26e05e97347, State: Initialized, Role: FOLLOWER
I20260620 19:07:48.572397 15330 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfafd442c86d474189fbe26e05e97347" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37253 } }
I20260620 19:07:48.572559 15330 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:07:48.572623 15330 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:07:48.572717 15330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:48.572995 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:07:48.574059 15330 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfafd442c86d474189fbe26e05e97347" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37253 } }
I20260620 19:07:48.574434 15330 leader_election.cc:302] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfafd442c86d474189fbe26e05e97347; no voters:
I20260620 19:07:48.574792 15330 leader_election.cc:288] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:07:48.575124 15335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:48.575584 15335 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [term 1 LEADER]: Becoming Leader. State: Replica: dfafd442c86d474189fbe26e05e97347, State: Running, Role: LEADER
I20260620 19:07:48.576011 15335 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfafd442c86d474189fbe26e05e97347" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37253 } }
I20260620 19:07:48.576108 15330 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:07:48.578491 15337 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfafd442c86d474189fbe26e05e97347. Latest consensus state: current_term: 1 leader_uuid: "dfafd442c86d474189fbe26e05e97347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfafd442c86d474189fbe26e05e97347" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37253 } } }
I20260620 19:07:48.578718 15337 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:48.578491 15336 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfafd442c86d474189fbe26e05e97347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfafd442c86d474189fbe26e05e97347" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37253 } } }
I20260620 19:07:48.578825 15336 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347 [sys.catalog]: This master's current role is: LEADER
W20260620 19:07:48.581300 15349 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:07:48.581375 15349 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:07:48.581460 15350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:07:48.582708 15350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:07:48.591171 15350 catalog_manager.cc:1347] Generated new cluster ID: 917a5cb25f4448048cfa8218f80476b2
I20260620 19:07:48.591446 15350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:07:48.617133 15350 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:07:48.618564 15350 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:07:48.634027 15350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfafd442c86d474189fbe26e05e97347: Generated new TSK 0
I20260620 19:07:48.634806 15350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:07:48.682770 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 15262
2026-06-20T19:07:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1547 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:07:48 dist-test-slave-6wrf krb5kdc[15357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:07:48 dist-test-slave-6wrf krb5kdc[15357](info): set up 2 sockets
Jun 20 19:07:48 dist-test-slave-6wrf krb5kdc[15357](info): commencing operation
krb5kdc: starting...
W20260620 19:07:49.783218 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:07:49 dist-test-slave-6wrf krb5kdc[15357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:07:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:07:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:07:49.921170 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43519
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:43519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:50.043830 15370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:50.044135 15370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:50.044181 15370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:50.048328 15370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:07:50.048398 15370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:50.048418 15370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:50.048435 15370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:07:50.048452 15370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:07:50.053992 15370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:43519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:50.055133 15370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:50.056206 15370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:50.064975 15375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:50.064977 15376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:50.064960 15378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:50.065207 15370 server_base.cc:1034] running on GCE node
I20260620 19:07:50.065979 15370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:50.067072 15370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:50.068375 15370 hybrid_clock.cc:648] HybridClock initialized: now 1781982470068356 us; error 55 us; skew 500 ppm
Jun 20 19:07:50 dist-test-slave-6wrf krb5kdc[15357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982470, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:50.076967 15370 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:07:50.078359 15370 webserver.cc:460] Webserver started at http://127.1.154.190:38325/ using document root <none> and password file <none>
I20260620 19:07:50.078999 15370 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:50.079071 15370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:50.079301 15370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:50.081439 15370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "6eb8583552164d88a308122bd7b04b2e"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "7a7726dc58227f7c9308541bfb26464e"
server_key_iv: "6a340960fb1863333ee57a92a5873dd4"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.082110 15370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "6eb8583552164d88a308122bd7b04b2e"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "7a7726dc58227f7c9308541bfb26464e"
server_key_iv: "6a340960fb1863333ee57a92a5873dd4"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.086189 15370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:07:50.089527 15385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.091564 15370 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:07:50.091810 15370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "6eb8583552164d88a308122bd7b04b2e"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "7a7726dc58227f7c9308541bfb26464e"
server_key_iv: "6a340960fb1863333ee57a92a5873dd4"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.091982 15370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:50.104080 15370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:50.108029 15370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:50.108281 15370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:50.116410 15370 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:43519
I20260620 19:07:50.116789 15437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:43519 every 8 connection(s)
I20260620 19:07:50.117489 15370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:07:50.119277 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 15370
I20260620 19:07:50.119549 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:07:50.119829 1642 external_mini_cluster.cc:1428] Setting key 505d0cf672085556b9227e31d10c6c64
I20260620 19:07:50.120493 15438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:50.128432 15438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: Bootstrap starting.
Jun 20 19:07:50 dist-test-slave-6wrf krb5kdc[15357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:50.132050 15438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:50.132864 15438 log.cc:824] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:50.134357 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.121217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.1.154.190:43519)
0620 19:07:50.121712 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:50.121727 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:50.121761 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:07:50.122452 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:50.122469 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:50.122777 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:50.123110 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:50.123911 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.124911 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.125817 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.126299 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.126481 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.129432 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:50.129437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:50.129453 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:50.129483 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:50.131473 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.131974 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.131979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.131983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.132092 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.132401 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.132404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.132406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.132805 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:50.133008 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:50.133305 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 19:07:50.133646 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":140,"threads_started":1}
I20260620 19:07:50.134990 15438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: No bootstrap required, opened a new log
I20260620 19:07:50.139159 15438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } }
I20260620 19:07:50.139464 15438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:50.139549 15438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eb8583552164d88a308122bd7b04b2e, State: Initialized, Role: FOLLOWER
I20260620 19:07:50.140074 15438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } }
I20260620 19:07:50.140256 15438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:07:50.140323 15438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:07:50.140431 15438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:50.141458 15438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } }
I20260620 19:07:50.141848 15438 leader_election.cc:302] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eb8583552164d88a308122bd7b04b2e; no voters:
I20260620 19:07:50.142203 15438 leader_election.cc:288] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:07:50.142383 15443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:50.142706 15443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 1 LEADER]: Becoming Leader. State: Replica: 6eb8583552164d88a308122bd7b04b2e, State: Running, Role: LEADER
I20260620 19:07:50.143004 15443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } }
I20260620 19:07:50.143523 15438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:07:50.144764 15444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6eb8583552164d88a308122bd7b04b2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } } }
I20260620 19:07:50.144912 15444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:50.145190 15445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eb8583552164d88a308122bd7b04b2e. Latest consensus state: current_term: 1 leader_uuid: "6eb8583552164d88a308122bd7b04b2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } } }
I20260620 19:07:50.145328 15445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:50.145418 15452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:07:50.148586 15452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:07:50.155514 15452 catalog_manager.cc:1347] Generated new cluster ID: 231a348e3ef24c5782d7a2003c69bf3c
I20260620 19:07:50.155643 15452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:07:50.174842 15452 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:07:50.176461 15452 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:07:50.188306 15452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: Generated new TSK 0
I20260620 19:07:50.189229 15452 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:07:50.221380 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43519
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:50.344578 15464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:50.344918 15464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:50.344993 15464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:50.349478 15464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:50.349581 15464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:50.349718 15464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:07:50.355228 15464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:50.356400 15464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:50.357445 15464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:50.368130 15470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:50.368067 15469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:50.368376 15464 server_base.cc:1034] running on GCE node
W20260620 19:07:50.368067 15472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:50.369500 15464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:50.370769 15464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:50.371986 15464 hybrid_clock.cc:648] HybridClock initialized: now 1781982470371940 us; error 63 us; skew 500 ppm
Jun 20 19:07:50 dist-test-slave-6wrf krb5kdc[15357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982470, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:50.375933 15464 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:07:50.376976 15464 webserver.cc:460] Webserver started at http://127.1.154.129:40851/ using document root <none> and password file <none>
I20260620 19:07:50.377679 15464 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:50.377772 15464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:50.377992 15464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:50.380157 15464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "ff6ce1bdaa824703b94d50083bfffbcc"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "997bef6c605b0668d686e6f9689e619d"
server_key_iv: "116556c4beb607782c9e54a5ca48d5de"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.380731 15464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "ff6ce1bdaa824703b94d50083bfffbcc"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "997bef6c605b0668d686e6f9689e619d"
server_key_iv: "116556c4beb607782c9e54a5ca48d5de"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.384510 15464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:07:50.387225 15479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.388494 15464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:07:50.388625 15464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "ff6ce1bdaa824703b94d50083bfffbcc"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "997bef6c605b0668d686e6f9689e619d"
server_key_iv: "116556c4beb607782c9e54a5ca48d5de"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.388753 15464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:50.407071 15464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:50.410419 15464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:50.410591 15464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:50.411259 15464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:50.412103 15464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:50.412153 15464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.412191 15464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:50.412245 15464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.422647 15464 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:40511
I20260620 19:07:50.422715 15592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:40511 every 8 connection(s)
I20260620 19:07:50.423636 15464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:07:50.429814 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 15464
I20260620 19:07:50.429958 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:07:50.430267 1642 external_mini_cluster.cc:1428] Setting key b351c5464a712c42fcacccd342b44bb7
Jun 20 19:07:50 dist-test-slave-6wrf krb5kdc[15357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982470, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
I20260620 19:07:50.442956 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.426529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:53997 (local address 127.1.154.190:43519)
0620 19:07:50.426807 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:50.426811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:50.427956 (+ 1145us) server_negotiation.cc:407] Connection header received
0620 19:07:50.429389 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:50.429392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:50.429455 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:50.429571 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:50.431247 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.432373 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.434755 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.435037 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.435315 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.437976 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:50.437980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:50.437983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:50.438019 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:50.439834 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.440618 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.440622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.440624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.440724 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.441263 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.441266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.441268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.441514 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:50.441675 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:50.442458 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 19:07:50.442767 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
I20260620 19:07:50.444519 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.427044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:53997)
0620 19:07:50.427758 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:50.427807 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:50.429106 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:50.429855 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:50.429877 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:50.430400 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:50.431107 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:50.431126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.432521 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.432541 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:50.434475 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:50.434496 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.435372 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.435378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:50.435596 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.436265 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:50.436282 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:50.437790 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:50.440051 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:50.440060 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:50.440089 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:50.440483 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:50.440842 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:50.440845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:50.440847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:50.441029 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:50.441709 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:50.441721 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:50.442115 (+ 394us) client_negotiation.cc:769] Sending connection context
0620 19:07:50.442393 (+ 278us) client_negotiation.cc:240] Negotiation successful
0620 19:07:50.443157 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":166,"threads_started":1}
I20260620 19:07:50.446478 15593 heartbeater.cc:347] Connected to a master server at 127.1.154.190:43519
I20260620 19:07:50.446861 15593 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:50.447603 15593 heartbeater.cc:511] Master 127.1.154.190:43519 requested a full tablet report, sending...
I20260620 19:07:50.449978 15402 ts_manager.cc:194] Registered new tserver with Master: ff6ce1bdaa824703b94d50083bfffbcc (127.1.154.129:40511)
I20260620 19:07:50.452152 15402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:53997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:07:50.465233 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43519
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:50.586174 15598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:50.586519 15598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:50.586566 15598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:50.590925 15598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:50.590994 15598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:50.591063 15598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:07:50.596700 15598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:50.598227 15598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:50.599241 15598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:50.611104 15604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:50.611130 15606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:50.611193 15598 server_base.cc:1034] running on GCE node
W20260620 19:07:50.611411 15603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:50.612109 15598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:50.613143 15598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:50.614396 15598 hybrid_clock.cc:648] HybridClock initialized: now 1781982470614347 us; error 65 us; skew 500 ppm
Jun 20 19:07:50 dist-test-slave-6wrf krb5kdc[15357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982470, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:50.617575 15598 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:07:50.618599 15598 webserver.cc:460] Webserver started at http://127.1.154.130:43477/ using document root <none> and password file <none>
I20260620 19:07:50.619184 15598 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:50.619275 15598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:50.619511 15598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:50.622881 15598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "cc97568fb1aa4aa89b6bb81caf384188"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "6464c1a63600962b5c7eb32a5dd3f91b"
server_key_iv: "7c1b92783a43d205940ea717f43bc0e5"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.623582 15598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "cc97568fb1aa4aa89b6bb81caf384188"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "6464c1a63600962b5c7eb32a5dd3f91b"
server_key_iv: "7c1b92783a43d205940ea717f43bc0e5"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.628705 15598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 19:07:50.632081 15613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.634212 15598 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:07:50.634424 15598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "cc97568fb1aa4aa89b6bb81caf384188"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "6464c1a63600962b5c7eb32a5dd3f91b"
server_key_iv: "7c1b92783a43d205940ea717f43bc0e5"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.634588 15598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:50.656030 15598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:50.659433 15598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:50.659650 15598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:50.660919 15598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:50.661964 15598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:50.662043 15598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.662113 15598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:50.662163 15598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.675433 15598 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:35689
I20260620 19:07:50.675460 15726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:35689 every 8 connection(s)
I20260620 19:07:50.676530 15598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:07:50.685479 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 15598
I20260620 19:07:50.685750 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:07:50.686003 1642 external_mini_cluster.cc:1428] Setting key 4e4eeb8c1c2abc017654990077f9d331
Jun 20 19:07:50 dist-test-slave-6wrf krb5kdc[15357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982470, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:50.695662 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.679040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:53153 (local address 127.1.154.190:43519)
0620 19:07:50.679281 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:50.679286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:50.681298 (+ 2012us) server_negotiation.cc:407] Connection header received
0620 19:07:50.682294 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:50.682298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:50.682370 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:50.682503 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:50.684296 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.684933 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.687288 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.687614 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.687736 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.691689 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:50.691693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:50.691695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:50.691720 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:50.693414 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.694066 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.694070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.694072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.694154 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.694507 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.694511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.694513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.694709 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:50.694818 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:50.695341 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 19:07:50.695488 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260620 19:07:50.696545 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.679659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:53153)
0620 19:07:50.681086 (+ 1427us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:50.681140 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:50.682029 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:50.682807 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:50.682818 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:50.683561 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:50.684090 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:50.684108 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.685338 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.685341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:50.687100 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:50.687112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.687829 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.687833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:50.688062 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.688936 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:50.688961 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:50.691472 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:50.693563 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:50.693570 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:50.693588 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:50.693954 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:50.694248 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:50.694252 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:50.694254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:50.694408 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:50.694816 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:50.694823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:50.695097 (+ 274us) client_negotiation.cc:769] Sending connection context
0620 19:07:50.695284 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 19:07:50.695650 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"thread_start_us":693,"threads_started":1}
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
I20260620 19:07:50.698048 15727 heartbeater.cc:347] Connected to a master server at 127.1.154.190:43519
I20260620 19:07:50.698673 15727 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:50.699489 15727 heartbeater.cc:511] Master 127.1.154.190:43519 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
I20260620 19:07:50.701318 15402 ts_manager.cc:194] Registered new tserver with Master: cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689)
I20260620 19:07:50.702277 15402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:53153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:07:50.720460 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43519
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:50.845974 15732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:50.846244 15732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:50.846289 15732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:50.852331 15732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:50.852428 15732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:50.852502 15732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:07:50.858747 15732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:50.859879 15732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:50.860919 15732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:50.871392 15737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:50.871376 15740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:50.871369 15738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:50.871771 15732 server_base.cc:1034] running on GCE node
I20260620 19:07:50.872300 15732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:50.873176 15732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:50.876530 15732 hybrid_clock.cc:648] HybridClock initialized: now 1781982470876501 us; error 67 us; skew 500 ppm
Jun 20 19:07:50 dist-test-slave-6wrf krb5kdc[15357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982470, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:50.879933 15732 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:07:50.880901 15732 webserver.cc:460] Webserver started at http://127.1.154.131:34155/ using document root <none> and password file <none>
I20260620 19:07:50.881482 15732 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:50.881544 15732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:50.881778 15732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:50.883836 15732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "6e5f44a58dd643bcb0f5666717572959"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "18c37a2a989321177cd9a5a3d001c87b"
server_key_iv: "f3471c0e966bab9d1e6c663f9d1906e0"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.884402 15732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "6e5f44a58dd643bcb0f5666717572959"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "18c37a2a989321177cd9a5a3d001c87b"
server_key_iv: "f3471c0e966bab9d1e6c663f9d1906e0"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.888293 15732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 19:07:50.891789 15747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.893960 15732 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:07:50.894120 15732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "6e5f44a58dd643bcb0f5666717572959"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "18c37a2a989321177cd9a5a3d001c87b"
server_key_iv: "f3471c0e966bab9d1e6c663f9d1906e0"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.894246 15732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:50.916028 15732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:50.921213 15732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:50.921528 15732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:50.922482 15732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:50.923725 15732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:50.923873 15732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.923947 15732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:50.924005 15732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.935720 15732 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:37383
I20260620 19:07:50.935803 15860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:37383 every 8 connection(s)
I20260620 19:07:50.937527 15732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:07:50.938807 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 15732
I20260620 19:07:50.938966 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:07:50.939399 1642 external_mini_cluster.cc:1428] Setting key 32e95000b2b90b3d56f38f89fa2be251
Jun 20 19:07:50 dist-test-slave-6wrf krb5kdc[15357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982470, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:50.956991 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.940725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:33049 (local address 127.1.154.190:43519)
0620 19:07:50.940864 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:50.940870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:50.943003 (+ 2133us) server_negotiation.cc:407] Connection header received
0620 19:07:50.944373 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:50.944375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:50.944428 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:50.944515 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:50.946482 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.947167 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.948639 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.948915 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.949011 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.951824 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:50.951829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:50.951832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:50.951878 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:50.954352 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.954964 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.954967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.954968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.955062 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.955492 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.955495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.955496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.955700 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:50.955778 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:50.956747 (+ 969us) server_negotiation.cc:299] Negotiation successful
0620 19:07:50.956868 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:07:50.958374 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.941351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:33049)
0620 19:07:50.942792 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:50.942875 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:50.943970 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:50.944820 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:50.944836 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:50.945399 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:50.946251 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:50.946270 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.947429 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.947434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:50.948475 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:50.948488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.949090 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.949094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:50.949250 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.949998 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:50.950016 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:50.951538 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:50.954469 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:50.954476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:50.954494 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:50.954814 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:50.955199 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:50.955202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:50.955204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:50.955402 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:50.955882 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:50.955892 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:50.956357 (+ 465us) client_negotiation.cc:769] Sending connection context
0620 19:07:50.956642 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 19:07:50.957070 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":780,"threads_started":1}
I20260620 19:07:50.959859 15861 heartbeater.cc:347] Connected to a master server at 127.1.154.190:43519
I20260620 19:07:50.960306 15861 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:50.961051 15861 heartbeater.cc:511] Master 127.1.154.190:43519 requested a full tablet report, sending...
I20260620 19:07:50.962939 15402 ts_manager.cc:194] Registered new tserver with Master: 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383)
I20260620 19:07:50.963761 15402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:33049
I20260620 19:07:50.968955 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:07:50.982172 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.971465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.1.154.190:43519)
0620 19:07:50.971708 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:50.971712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:50.971732 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:07:50.971913 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:50.971917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:50.971973 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:50.972088 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:50.973236 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.973893 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.974870 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.975357 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.975511 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.977501 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:50.977505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:50.977508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:50.977541 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:50.979573 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.980194 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.980198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.980200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.980358 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.980858 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.980861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.980863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.981275 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:50.981414 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:50.981861 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 19:07:50.982026 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 19:07:50.987202 15402 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:07:50.990782 15402 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:07:51.007256 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.001708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.1.154.131:37383)
0620 19:07:51.002132 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:51.002142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:51.002162 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:07:51.002238 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:51.002243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:51.002413 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:51.002549 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:51.002965 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.003761 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.005755 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.006450 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.006575 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.006720 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:51.006986 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 19:07:51.007088 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":123,"threads_started":1}
I20260620 19:07:51.007267 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.001227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60072 (local address 127.1.154.129:40511)
0620 19:07:51.001485 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:51.001494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:51.001784 (+ 290us) server_negotiation.cc:407] Connection header received
0620 19:07:51.002032 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:51.002038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:51.002236 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:51.002350 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:51.002850 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.004144 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.005648 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.006382 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.006550 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.006719 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:51.007028 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 19:07:51.007119 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":93,"threads_started":1}
I20260620 19:07:51.007256 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.000345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:40511 (local address 127.0.0.1:60072)
0620 19:07:51.001680 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:51.001694 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:51.001866 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:51.002550 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:51.002556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:51.002577 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:51.002714 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.002721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.004304 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.004307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.005492 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.005500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.006558 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.006562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.006788 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.006841 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 19:07:51.006932 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 19:07:51.007088 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1241,"mutex_wait_us":116,"thread_start_us":115,"threads_started":1}
I20260620 19:07:51.007259 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.001383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:37383 (local address 127.0.0.1:41944)
0620 19:07:51.002051 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:51.002066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:51.002147 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:51.002691 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:51.002694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:51.002707 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:51.002849 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.002856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.003895 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.003907 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.005634 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.005643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.006576 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.006579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.006768 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.006823 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 19:07:51.006953 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 19:07:51.007090 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":167,"spinlock_wait_cycles":768,"thread_start_us":123,"threads_started":1}
I20260620 19:07:51.008922 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.000301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:35689 (local address 127.0.0.1:60516)
0620 19:07:51.000961 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:51.001026 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:51.001164 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:51.002696 (+ 1532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:51.002699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:51.002718 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:51.002883 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.002890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.004284 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.004288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.005488 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.005498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.008479 (+ 2981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.008483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.008571 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.008615 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 19:07:51.008713 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 19:07:51.008786 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":37,"thread_start_us":346,"threads_started":1}
I20260620 19:07:51.008965 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.000678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.1.154.130:35689)
0620 19:07:51.001311 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:51.001327 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:51.001363 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:07:51.001486 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:51.001496 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:51.002499 (+ 1003us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:51.002708 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:51.003048 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.004144 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.007512 (+ 3368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.008332 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.008467 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.008603 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:51.008741 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 19:07:51.008831 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":124,"threads_started":1}
I20260620 19:07:51.011761 15527 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2a174dee7f4eeeb1403d9f823375dd (DEFAULT_TABLE table=test-table [id=c675118a37054c64a077efacb6b0d0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:51.012056 15795 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2a174dee7f4eeeb1403d9f823375dd (DEFAULT_TABLE table=test-table [id=c675118a37054c64a077efacb6b0d0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:51.012290 15661 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2a174dee7f4eeeb1403d9f823375dd (DEFAULT_TABLE table=test-table [id=c675118a37054c64a077efacb6b0d0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:51.012957 15527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2a174dee7f4eeeb1403d9f823375dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:51.013165 15661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2a174dee7f4eeeb1403d9f823375dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:51.013509 15795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2a174dee7f4eeeb1403d9f823375dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:51.020443 15878 tablet_bootstrap.cc:492] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959: Bootstrap starting.
I20260620 19:07:51.020879 15879 tablet_bootstrap.cc:492] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: Bootstrap starting.
I20260620 19:07:51.023372 15879 tablet_bootstrap.cc:654] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:51.023904 15880 tablet_bootstrap.cc:492] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: Bootstrap starting.
I20260620 19:07:51.023975 15878 tablet_bootstrap.cc:654] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:51.024385 15879 log.cc:824] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:51.024717 15878 log.cc:824] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:51.026435 15880 tablet_bootstrap.cc:654] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:51.026796 15878 tablet_bootstrap.cc:492] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959: No bootstrap required, opened a new log
I20260620 19:07:51.026851 15879 tablet_bootstrap.cc:492] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: No bootstrap required, opened a new log
I20260620 19:07:51.027009 15878 ts_tablet_manager.cc:1397] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 19:07:51.027040 15879 ts_tablet_manager.cc:1397] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:07:51.027267 15880 log.cc:824] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:51.029495 15880 tablet_bootstrap.cc:492] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: No bootstrap required, opened a new log
I20260620 19:07:51.029731 15880 ts_tablet_manager.cc:1397] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:07:51.030340 15878 raft_consensus.cc:348] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } }
I20260620 19:07:51.030596 15878 raft_consensus.cc:374] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:51.030645 15878 raft_consensus.cc:729] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5f44a58dd643bcb0f5666717572959, State: Initialized, Role: FOLLOWER
I20260620 19:07:51.031097 15878 consensus_queue.cc:260] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } }
I20260620 19:07:51.031759 15861 heartbeater.cc:503] Master 127.1.154.190:43519 was elected leader, sending a full tablet report...
I20260620 19:07:51.031777 15879 raft_consensus.cc:348] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } }
I20260620 19:07:51.032042 15879 raft_consensus.cc:374] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:51.032094 15879 raft_consensus.cc:729] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6ce1bdaa824703b94d50083bfffbcc, State: Initialized, Role: FOLLOWER
I20260620 19:07:51.032100 15878 ts_tablet_manager.cc:1428] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:07:51.032524 15879 consensus_queue.cc:260] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } }
I20260620 19:07:51.033510 15879 ts_tablet_manager.cc:1428] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:07:51.033916 15593 heartbeater.cc:503] Master 127.1.154.190:43519 was elected leader, sending a full tablet report...
I20260620 19:07:51.033864 15880 raft_consensus.cc:348] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } }
I20260620 19:07:51.034173 15880 raft_consensus.cc:374] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:51.034235 15880 raft_consensus.cc:729] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc97568fb1aa4aa89b6bb81caf384188, State: Initialized, Role: FOLLOWER
I20260620 19:07:51.034657 15880 consensus_queue.cc:260] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } }
I20260620 19:07:51.035429 15727 heartbeater.cc:503] Master 127.1.154.190:43519 was elected leader, sending a full tablet report...
I20260620 19:07:51.035620 15880 ts_tablet_manager.cc:1428] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:07:51.129760 15886 raft_consensus.cc:482] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:51.130026 15886 raft_consensus.cc:504] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } }
I20260620 19:07:51.131734 15886 leader_election.cc:288] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff6ce1bdaa824703b94d50083bfffbcc (127.1.154.129:40511), 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383)
I20260620 19:07:51.136945 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.132130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:40511 (local address 127.1.154.130:51907)
0620 19:07:51.132380 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:51.132400 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:51.132644 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:51.133189 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:51.133194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:51.133238 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:51.133695 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.133706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.134814 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.134818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.135720 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.135729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.136439 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.136450 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.136577 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.136634 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 19:07:51.136696 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:07:51.136788 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":166,"spinlock_wait_cycles":12672}
I20260620 19:07:51.136979 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.132254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:51907 (local address 127.1.154.129:40511)
0620 19:07:51.132396 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:51.132401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:51.132571 (+ 170us) server_negotiation.cc:407] Connection header received
0620 19:07:51.132882 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:51.132886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:51.132938 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:51.133053 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:51.133878 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.134680 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.135862 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.136302 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.136450 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.136515 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:51.136747 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 19:07:51.136840 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:07:51.137014 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.132183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:37383 (local address 127.1.154.130:51933)
0620 19:07:51.132501 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:51.132513 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:51.132644 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:51.133189 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:51.133194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:51.133233 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:51.133698 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.133706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.134862 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.134865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.135723 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.135732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.136447 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.136451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.136612 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.136696 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 19:07:51.136794 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 19:07:51.136864 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":277,"spinlock_wait_cycles":4096,"thread_start_us":90,"threads_started":1}
I20260620 19:07:51.137025 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.132256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:51933 (local address 127.1.154.131:37383)
0620 19:07:51.132413 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:51.132420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:51.132808 (+ 388us) server_negotiation.cc:407] Connection header received
0620 19:07:51.132879 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:51.132882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:51.132938 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:51.133053 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:51.133896 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.134678 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.135904 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.136303 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.136438 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.136491 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:51.136821 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 19:07:51.136893 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 19:07:51.137833 15547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2a174dee7f4eeeb1403d9f823375dd" candidate_uuid: "cc97568fb1aa4aa89b6bb81caf384188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" is_pre_election: true
I20260620 19:07:51.138226 15547 raft_consensus.cc:2454] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc97568fb1aa4aa89b6bb81caf384188 in term 0.
I20260620 19:07:51.138546 15815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2a174dee7f4eeeb1403d9f823375dd" candidate_uuid: "cc97568fb1aa4aa89b6bb81caf384188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5f44a58dd643bcb0f5666717572959" is_pre_election: true
I20260620 19:07:51.138819 15815 raft_consensus.cc:2454] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc97568fb1aa4aa89b6bb81caf384188 in term 0.
I20260620 19:07:51.139243 15614 leader_election.cc:302] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6e5f44a58dd643bcb0f5666717572959, cc97568fb1aa4aa89b6bb81caf384188, ff6ce1bdaa824703b94d50083bfffbcc; no voters:
I20260620 19:07:51.139554 15886 raft_consensus.cc:2781] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:07:51.139637 15886 raft_consensus.cc:482] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:07:51.139703 15886 raft_consensus.cc:3037] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:51.140842 15886 raft_consensus.cc:504] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } }
I20260620 19:07:51.141264 15886 leader_election.cc:288] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [CANDIDATE]: Term 1 election: Requested vote from peers ff6ce1bdaa824703b94d50083bfffbcc (127.1.154.129:40511), 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383)
I20260620 19:07:51.141834 15815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2a174dee7f4eeeb1403d9f823375dd" candidate_uuid: "cc97568fb1aa4aa89b6bb81caf384188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5f44a58dd643bcb0f5666717572959"
I20260620 19:07:51.141938 15547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2a174dee7f4eeeb1403d9f823375dd" candidate_uuid: "cc97568fb1aa4aa89b6bb81caf384188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6ce1bdaa824703b94d50083bfffbcc"
I20260620 19:07:51.142091 15547 raft_consensus.cc:3037] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:51.142076 15815 raft_consensus.cc:3037] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:51.143893 15815 raft_consensus.cc:2454] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc97568fb1aa4aa89b6bb81caf384188 in term 1.
I20260620 19:07:51.143893 15547 raft_consensus.cc:2454] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc97568fb1aa4aa89b6bb81caf384188 in term 1.
I20260620 19:07:51.144446 15614 leader_election.cc:302] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc97568fb1aa4aa89b6bb81caf384188, ff6ce1bdaa824703b94d50083bfffbcc; no voters:
I20260620 19:07:51.144714 15886 raft_consensus.cc:2781] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:51.145008 15886 raft_consensus.cc:686] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 1 LEADER]: Becoming Leader. State: Replica: cc97568fb1aa4aa89b6bb81caf384188, State: Running, Role: LEADER
I20260620 19:07:51.145457 15886 consensus_queue.cc:237] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } }
I20260620 19:07:51.149372 15398 catalog_manager.cc:5495] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130). New cstate: current_term: 1 leader_uuid: "cc97568fb1aa4aa89b6bb81caf384188" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260620 19:07:51.170118 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.165835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.1.154.130:35689)
0620 19:07:51.166020 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:51.166023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:51.166038 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:07:51.166117 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:51.166119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:51.166176 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:51.166275 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:07:51.166554 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.167052 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.167951 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.168204 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.168291 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.168780 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:07:51.169499 (+ 719us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:07:51.169744 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:51.169789 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 19:07:51.169896 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260620 19:07:51.175246 15594 tablet.cc:2369] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:07:51.177637 15728 tablet.cc:2369] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:07:51.177747 15815 raft_consensus.cc:1261] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959 [term 1 FOLLOWER]: Refusing update from remote peer cc97568fb1aa4aa89b6bb81caf384188: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:51.177852 15547 raft_consensus.cc:1261] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Refusing update from remote peer cc97568fb1aa4aa89b6bb81caf384188: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:51.178706 15886 consensus_queue.cc:1035] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:51.178982 15888 consensus_queue.cc:1035] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:51.186270 15891 mvcc.cc:205] Tried to move back new op lower bound from 7299000201933983744 to 7299000201814786048. Current Snapshot: MvccSnapshot[applied={T|T < 7299000201933983744}]
I20260620 19:07:51.196532 15398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:51.196882 15398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:51.199882 15398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: Sending DeleteTablet for 3 replicas of tablet 0e2a174dee7f4eeeb1403d9f823375dd
I20260620 19:07:51.201431 15661 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2a174dee7f4eeeb1403d9f823375dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:51 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:60516
I20260620 19:07:51.201431 15527 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2a174dee7f4eeeb1403d9f823375dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:51 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:60072
I20260620 19:07:51.202093 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 15464
I20260620 19:07:51.203091 15795 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2a174dee7f4eeeb1403d9f823375dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:51 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:41944
I20260620 19:07:51.203683 15902 tablet_replica.cc:330] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959: stopping tablet replica
I20260620 19:07:51.204222 15902 raft_consensus.cc:2229] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:51.204698 15902 raft_consensus.cc:2258] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:51.207034 15902 ts_tablet_manager.cc:1905] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:51.209672 15901 tablet_replica.cc:330] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: stopping tablet replica
W20260620 19:07:51.209926 15386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:40511 (error 108)
I20260620 19:07:51.210031 15901 raft_consensus.cc:2229] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:07:51.210235 15902 ts_tablet_manager.cc:1918] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:07:51.210345 15902 log.cc:1196] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/0e2a174dee7f4eeeb1403d9f823375dd
I20260620 19:07:51.210486 15901 raft_consensus.cc:2258] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:51.210693 15902 ts_tablet_manager.cc:1939] T 0e2a174dee7f4eeeb1403d9f823375dd P 6e5f44a58dd643bcb0f5666717572959: Deleting consensus metadata
I20260620 19:07:51.211400 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 15598
I20260620 19:07:51.211418 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.211080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:40511 (local address 127.0.0.1:60084)
0620 19:07:51.211228 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:51.211324 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 19:07:51.211967 15386 catalog_manager.cc:4584] TS ff6ce1bdaa824703b94d50083bfffbcc (127.1.154.129:40511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e2a174dee7f4eeeb1403d9f823375dd: Network error: Client connection negotiation failed: client connection to 127.1.154.129:40511: connect: Connection refused (error 111)
I20260620 19:07:51.213277 15388 catalog_manager.cc:4841] TS 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383): tablet 0e2a174dee7f4eeeb1403d9f823375dd (table test-table [id=c675118a37054c64a077efacb6b0d0bc]) successfully deleted
I20260620 19:07:51.218166 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 15732
I20260620 19:07:51.220178 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.219589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:35689 (local address 127.0.0.1:60538)
0620 19:07:51.219920 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:51.220028 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.130:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260620 19:07:51.226220 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 15370
I20260620 19:07:51.234402 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43519
--webserver_interface=127.1.154.190
--webserver_port=38325
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:43519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:51.359932 15904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:51.360229 15904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:51.360271 15904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:51.366298 15904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:07:51.366391 15904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:51.366412 15904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:51.366432 15904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:07:51.366448 15904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:07:51.374519 15904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:43519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=38325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:51.376253 15904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:51.377425 15904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:51.393527 15911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:51.393527 15913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:51.393834 15904 server_base.cc:1034] running on GCE node
W20260620 19:07:51.393519 15910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:51.394582 15904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:51.395850 15904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:51.397071 15904 hybrid_clock.cc:648] HybridClock initialized: now 1781982471397047 us; error 53 us; skew 500 ppm
Jun 20 19:07:51 dist-test-slave-6wrf krb5kdc[15357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982471, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:51.400089 15904 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:07:51.401044 15904 webserver.cc:460] Webserver started at http://127.1.154.190:38325/ using document root <none> and password file <none>
I20260620 19:07:51.401659 15904 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:51.401750 15904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:51.406370 15904 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 19:07:51.409437 15920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:51.411512 15904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:07:51.411741 15904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "6eb8583552164d88a308122bd7b04b2e"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "7a7726dc58227f7c9308541bfb26464e"
server_key_iv: "6a340960fb1863333ee57a92a5873dd4"
server_key_version: "encryptionkey@0"
I20260620 19:07:51.412770 15904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:51.428937 15904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:51.432724 15904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:51.432955 15904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:51.441192 15904 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:43519
I20260620 19:07:51.441519 15972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:43519 every 8 connection(s)
I20260620 19:07:51.442770 15904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:07:51.448402 15973 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:07:51.450309 15973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: Bootstrap starting.
I20260620 19:07:51.452831 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 15904
I20260620 19:07:51.453354 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:40511
--local_ip_for_outbound_sockets=127.1.154.129
--tserver_master_addrs=127.1.154.190:43519
--webserver_port=40851
--webserver_interface=127.1.154.129
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:07:51.471388 15973 log.cc:824] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:51.482132 15973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:07:51.482568 15973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: Bootstrap complete.
I20260620 19:07:51.486841 15973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } }
I20260620 19:07:51.487257 15973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eb8583552164d88a308122bd7b04b2e, State: Initialized, Role: FOLLOWER
I20260620 19:07:51.487782 15973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } }
I20260620 19:07:51.487922 15973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:07:51.488039 15973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:07:51.488166 15973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:07:51.490300 15973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } }
I20260620 19:07:51.490657 15973 leader_election.cc:302] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eb8583552164d88a308122bd7b04b2e; no voters:
I20260620 19:07:51.491020 15973 leader_election.cc:288] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:07:51.491113 15977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:07:51.491348 15977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 2 LEADER]: Becoming Leader. State: Replica: 6eb8583552164d88a308122bd7b04b2e, State: Running, Role: LEADER
I20260620 19:07:51.491636 15977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } }
I20260620 19:07:51.492301 15973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:07:51.493090 15978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6eb8583552164d88a308122bd7b04b2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } } }
I20260620 19:07:51.493214 15978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:51.493580 15979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eb8583552164d88a308122bd7b04b2e. Latest consensus state: current_term: 2 leader_uuid: "6eb8583552164d88a308122bd7b04b2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } } }
I20260620 19:07:51.493685 15979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:51.494086 15983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:07:51.498816 15983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:07:51.499467 15983 catalog_manager.cc:1259] Loaded cluster ID: 231a348e3ef24c5782d7a2003c69bf3c
I20260620 19:07:51.499553 15983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:07:51.500640 15983 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 19:07:51.503520 15983 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 19:07:51.508291 15983 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:07:51.508960 15983 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: Loaded TSK: 0
I20260620 19:07:51.510021 15983 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:07:51.582926 15975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:51.583243 15975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:51.583323 15975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:51.588937 15975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:51.589028 15975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:51.589125 15975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:07:51.597069 15975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:40511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=40851
--tserver_master_addrs=127.1.154.190:43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:51.598347 15975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:51.599375 15975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:07:51.612290 15975 server_base.cc:1034] running on GCE node
W20260620 19:07:51.612226 16003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:51.612514 16001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:51.612227 16000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:51.613026 15975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:51.613787 15975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:51.615005 15975 hybrid_clock.cc:648] HybridClock initialized: now 1781982471614996 us; error 44 us; skew 500 ppm
Jun 20 19:07:51 dist-test-slave-6wrf krb5kdc[15357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982471, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:51.618311 15975 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:07:51.619243 15975 webserver.cc:460] Webserver started at http://127.1.154.129:40851/ using document root <none> and password file <none>
I20260620 19:07:51.619800 15975 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:51.619905 15975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:51.625288 15975 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:07:51.627998 16010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:51.629468 15975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 19:07:51.629581 15975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "ff6ce1bdaa824703b94d50083bfffbcc"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "997bef6c605b0668d686e6f9689e619d"
server_key_iv: "116556c4beb607782c9e54a5ca48d5de"
server_key_version: "encryptionkey@0"
I20260620 19:07:51.630330 15975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:51.677870 15975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:51.682390 15975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:51.682619 15975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:51.683293 15975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:51.684544 16017 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:07:51.688699 15975 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:07:51.688791 15975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 19:07:51.688843 15975 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:07:51.690629 15975 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:07:51.690675 15975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260620 19:07:51.690969 16017 tablet_bootstrap.cc:492] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: Bootstrap starting.
I20260620 19:07:51.701414 15975 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:40511
I20260620 19:07:51.701433 16124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:40511 every 8 connection(s)
I20260620 19:07:51.702502 15975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:07:51.711954 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 15975
I20260620 19:07:51.712446 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:35689
--local_ip_for_outbound_sockets=127.1.154.130
--tserver_master_addrs=127.1.154.190:43519
--webserver_port=43477
--webserver_interface=127.1.154.130
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:07:51.713080 16017 log.cc:824] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:07:51 dist-test-slave-6wrf krb5kdc[15357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982471, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:51.721210 16017 tablet_bootstrap.cc:492] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:07:51.721726 16017 tablet_bootstrap.cc:492] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: Bootstrap complete.
I20260620 19:07:51.723019 16017 ts_tablet_manager.cc:1397] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.004s
I20260620 19:07:51.724752 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.704986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:37603)
0620 19:07:51.705582 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:51.705672 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:51.706837 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:51.708802 (+ 1965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:51.708811 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:51.709155 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:51.709666 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.709692 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.711781 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.711785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.713249 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.713264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.714400 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.714404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.714588 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.715309 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:51.715329 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:51.717505 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:51.721036 (+ 3531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:51.721042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:51.721054 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:51.721444 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:51.721960 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:51.721963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:51.721965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:51.722134 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:51.723003 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:51.723010 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:51.723293 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 19:07:51.723479 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 19:07:51.723827 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":183,"threads_started":1}
I20260620 19:07:51.725024 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.705008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:37603 (local address 127.1.154.190:43519)
0620 19:07:51.707495 (+ 2487us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:51.707521 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:51.707578 (+ 57us) server_negotiation.cc:407] Connection header received
0620 19:07:51.707909 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:51.707924 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:51.708364 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:51.708754 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:51.709863 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.711575 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.713397 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.714207 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.714409 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.717835 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:51.717845 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:51.717864 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:51.717911 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:51.720881 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:51.721629 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:51.721638 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:51.721645 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:51.721842 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:51.722264 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:51.722270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:51.722275 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:51.722847 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:51.723036 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:51.723677 (+ 641us) server_negotiation.cc:299] Negotiation successful
0620 19:07:51.724047 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2181,"thread_start_us":113,"threads_started":1}
I20260620 19:07:51.726063 16017 raft_consensus.cc:348] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } }
I20260620 19:07:51.726579 16017 raft_consensus.cc:729] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6ce1bdaa824703b94d50083bfffbcc, State: Initialized, Role: FOLLOWER
I20260620 19:07:51.727027 16017 consensus_queue.cc:260] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } }
I20260620 19:07:51.728001 16017 ts_tablet_manager.cc:1428] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:07:51.728149 16125 heartbeater.cc:347] Connected to a master server at 127.1.154.190:43519
I20260620 19:07:51.728363 16125 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:51.728852 16125 heartbeater.cc:511] Master 127.1.154.190:43519 requested a full tablet report, sending...
I20260620 19:07:51.731828 15937 ts_manager.cc:194] Registered new tserver with Master: ff6ce1bdaa824703b94d50083bfffbcc (127.1.154.129:40511)
I20260620 19:07:51.736151 15937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:37603
I20260620 19:07:51.740141 16125 heartbeater.cc:503] Master 127.1.154.190:43519 was elected leader, sending a full tablet report...
Jun 20 19:07:51 dist-test-slave-6wrf krb5kdc[15357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982471, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for kudu/127.1.154.129@KRBTEST.COM
I20260620 19:07:51.751403 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.735968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:40511 (local address 127.0.0.1:60088)
0620 19:07:51.736954 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:51.737023 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:51.737304 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:51.738411 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:51.738419 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:51.739234 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:51.739430 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.739442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.740994 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.740999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.742052 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.742066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.742990 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.742994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.743157 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.743981 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:51.743996 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:51.746105 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:51.748708 (+ 2603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:51.748719 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:51.748726 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:51.749020 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:51.749353 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:51.749358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:51.749359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:51.749457 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:51.750330 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:51.750340 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:51.750576 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 19:07:51.750779 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 19:07:51.751155 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":208,"threads_started":1}
I20260620 19:07:51.751403 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.735982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60088 (local address 127.1.154.129:40511)
0620 19:07:51.736602 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:51.736617 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:51.737146 (+ 529us) server_negotiation.cc:407] Connection header received
0620 19:07:51.737740 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:51.737749 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:51.738002 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:51.738183 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:51.739642 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.740819 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.742235 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.742830 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.742974 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.746472 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:51.746487 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:51.746500 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:51.746535 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:51.748548 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:51.749156 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:51.749169 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:51.749170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:51.749252 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:51.749554 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:51.749558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:51.749560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:51.749879 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:51.750375 (+ 496us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:51.750912 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 19:07:51.751155 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":199,"threads_started":1}
I20260620 19:07:51.753291 16059 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2a174dee7f4eeeb1403d9f823375dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:51 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:60088
I20260620 19:07:51.754770 16134 tablet_replica.cc:330] stopping tablet replica
I20260620 19:07:51.755119 16134 raft_consensus.cc:2229] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:51.755383 16134 raft_consensus.cc:2258] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:51.756369 16134 ts_tablet_manager.cc:1905] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:51.759933 16134 ts_tablet_manager.cc:1918] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:07:51.760015 16134 log.cc:1196] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal/wals/0e2a174dee7f4eeeb1403d9f823375dd
I20260620 19:07:51.760277 16134 ts_tablet_manager.cc:1939] T 0e2a174dee7f4eeeb1403d9f823375dd P ff6ce1bdaa824703b94d50083bfffbcc: Deleting consensus metadata
I20260620 19:07:51.762171 15921 catalog_manager.cc:4841] TS ff6ce1bdaa824703b94d50083bfffbcc (127.1.154.129:40511): tablet 0e2a174dee7f4eeeb1403d9f823375dd (table test-table [id=c675118a37054c64a077efacb6b0d0bc]) successfully deleted
W20260620 19:07:51.843910 16130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:51.844254 16130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:51.844328 16130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:51.848676 16130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:51.848784 16130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:51.848881 16130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:07:51.854477 16130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:35689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=43477
--tserver_master_addrs=127.1.154.190:43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:51.855620 16130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:51.856671 16130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:51.866801 16140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:51.866933 16142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:51.866801 16139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:51.866982 16130 server_base.cc:1034] running on GCE node
I20260620 19:07:51.867786 16130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:51.868510 16130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:51.869793 16130 hybrid_clock.cc:648] HybridClock initialized: now 1781982471869777 us; error 36 us; skew 500 ppm
Jun 20 19:07:51 dist-test-slave-6wrf krb5kdc[15357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982471, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:51.872819 16130 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:07:51.873862 16130 webserver.cc:460] Webserver started at http://127.1.154.130:43477/ using document root <none> and password file <none>
I20260620 19:07:51.874399 16130 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:51.874483 16130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:51.878432 16130 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 19:07:51.881206 16149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:51.882798 16130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:07:51.882929 16130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "cc97568fb1aa4aa89b6bb81caf384188"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "6464c1a63600962b5c7eb32a5dd3f91b"
server_key_iv: "7c1b92783a43d205940ea717f43bc0e5"
server_key_version: "encryptionkey@0"
I20260620 19:07:51.883527 16130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:51.895653 16130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:51.899240 16130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:51.899469 16130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:51.900118 16130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:51.901350 16156 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:07:51.905542 16130 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:07:51.905663 16130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 19:07:51.905769 16130 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:07:51.907588 16130 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:07:51.907637 16130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 19:07:51.908298 16156 tablet_bootstrap.cc:492] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: Bootstrap starting.
I20260620 19:07:51.916519 16156 log.cc:824] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:51.918576 16130 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:35689
I20260620 19:07:51.918645 16264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:35689 every 8 connection(s)
I20260620 19:07:51.919606 16130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:07:51.921206 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 16130
I20260620 19:07:51.921758 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:37383
--local_ip_for_outbound_sockets=127.1.154.131
--tserver_master_addrs=127.1.154.190:43519
--webserver_port=34155
--webserver_interface=127.1.154.131
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:07:51.925987 16156 tablet_bootstrap.cc:492] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:07:51.926889 16156 tablet_bootstrap.cc:492] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: Bootstrap complete.
I20260620 19:07:51.927562 16156 ts_tablet_manager.cc:1397] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.005s
Jun 20 19:07:51 dist-test-slave-6wrf krb5kdc[15357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982471, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:51.931879 16156 raft_consensus.cc:348] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } }
I20260620 19:07:51.932546 16156 raft_consensus.cc:729] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc97568fb1aa4aa89b6bb81caf384188, State: Initialized, Role: FOLLOWER
I20260620 19:07:51.933136 16156 consensus_queue.cc:260] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } }
I20260620 19:07:51.934098 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.921669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:55121 (local address 127.1.154.190:43519)
0620 19:07:51.921813 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:51.921818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:51.922528 (+ 710us) server_negotiation.cc:407] Connection header received
0620 19:07:51.923973 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:51.923977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:51.924090 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:51.924184 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:51.925279 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.925911 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.927128 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.927367 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.927489 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.930423 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:51.930426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:51.930428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:51.930449 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:51.931859 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:51.932364 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:51.932369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:51.932371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:51.932450 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:51.932936 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:51.932942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:51.932946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:51.933131 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:51.933241 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:51.933818 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 19:07:51.933956 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:07:51.934376 16156 ts_tablet_manager.cc:1428] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 19:07:51.935459 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.921857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:55121)
0620 19:07:51.922376 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:51.922428 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:51.923757 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:51.924342 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:51.924349 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:51.924696 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:51.925137 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.925150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.926108 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.926111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.927001 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.927015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.927501 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.927505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.927682 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.928360 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:51.928375 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:51.930248 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:51.931991 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:51.931996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:51.932010 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:51.932260 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:51.932610 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:51.932612 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:51.932613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:51.932796 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:51.933240 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:51.933246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:51.933512 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 19:07:51.934278 (+ 766us) client_negotiation.cc:240] Negotiation successful
0620 19:07:51.934647 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":136,"threads_started":1}
I20260620 19:07:51.936589 16265 heartbeater.cc:347] Connected to a master server at 127.1.154.190:43519
I20260620 19:07:51.936832 16265 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:51.937304 16265 heartbeater.cc:511] Master 127.1.154.190:43519 requested a full tablet report, sending...
I20260620 19:07:51.939248 15937 ts_manager.cc:194] Registered new tserver with Master: cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689)
I20260620 19:07:51.940866 15937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:55121
I20260620 19:07:51.943815 16265 heartbeater.cc:503] Master 127.1.154.190:43519 was elected leader, sending a full tablet report...
Jun 20 19:07:51 dist-test-slave-6wrf krb5kdc[15357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982471, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for kudu/127.1.154.130@KRBTEST.COM
I20260620 19:07:51.952059 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.940407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:35689 (local address 127.0.0.1:60548)
0620 19:07:51.940579 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:51.940594 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:51.940669 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:51.941778 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:51.941782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:51.942027 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:51.942157 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.942168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.943754 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.943758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.944563 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.944571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.946339 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.946343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.946435 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.946997 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:51.947011 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:51.948292 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:51.950355 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:51.950358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:51.950361 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:51.950564 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:51.950869 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:51.950873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:51.950875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:51.950964 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:51.951445 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:51.951449 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:51.951593 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 19:07:51.951711 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 19:07:51.951879 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 19:07:51.952129 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.940661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.1.154.130:35689)
0620 19:07:51.941212 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:51.941226 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:51.941255 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:07:51.941327 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:51.941334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:51.941575 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:51.941770 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:51.942314 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.943587 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.944699 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.945095 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.945235 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.948440 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:51.948450 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:51.948464 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:51.948494 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:51.950217 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:51.950681 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:51.950686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:51.950688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:51.950773 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:51.951054 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:51.951061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:51.951065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:51.951328 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:51.951476 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:51.951794 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 19:07:51.951961 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":424,"thread_start_us":185,"threads_started":1}
I20260620 19:07:51.953217 16198 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2a174dee7f4eeeb1403d9f823375dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:51 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:60548
I20260620 19:07:51.954547 16271 tablet_replica.cc:330] stopping tablet replica
I20260620 19:07:51.955034 16271 raft_consensus.cc:2229] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:51.955408 16271 raft_consensus.cc:2258] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:51.956905 16271 ts_tablet_manager.cc:1905] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:51.962059 16271 ts_tablet_manager.cc:1918] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:07:51.962225 16271 log.cc:1196] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/0e2a174dee7f4eeeb1403d9f823375dd
I20260620 19:07:51.962803 16271 ts_tablet_manager.cc:1939] T 0e2a174dee7f4eeeb1403d9f823375dd P cc97568fb1aa4aa89b6bb81caf384188: Deleting consensus metadata
I20260620 19:07:51.964744 15924 catalog_manager.cc:4841] TS cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689): tablet 0e2a174dee7f4eeeb1403d9f823375dd (table test-table [id=c675118a37054c64a077efacb6b0d0bc]) successfully deleted
W20260620 19:07:52.051970 16268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:52.052242 16268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:52.052287 16268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:52.056533 16268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:52.056619 16268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:52.056684 16268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:07:52.062109 16268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:37383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=34155
--tserver_master_addrs=127.1.154.190:43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:52.063369 16268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:52.064330 16268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:52.074859 16277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:52.074883 16276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:52.074859 16279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:52.075579 16268 server_base.cc:1034] running on GCE node
I20260620 19:07:52.075950 16268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:52.076617 16268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:52.079901 16268 hybrid_clock.cc:648] HybridClock initialized: now 1781982472079834 us; error 87 us; skew 500 ppm
Jun 20 19:07:52 dist-test-slave-6wrf krb5kdc[15357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982472, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:52.082912 16268 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:07:52.083818 16268 webserver.cc:460] Webserver started at http://127.1.154.131:34155/ using document root <none> and password file <none>
I20260620 19:07:52.084362 16268 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:52.084438 16268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:52.089134 16268 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 19:07:52.092235 16286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:52.094079 16268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:07:52.094198 16268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "6e5f44a58dd643bcb0f5666717572959"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "18c37a2a989321177cd9a5a3d001c87b"
server_key_iv: "f3471c0e966bab9d1e6c663f9d1906e0"
server_key_version: "encryptionkey@0"
I20260620 19:07:52.094861 16268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:52.115767 16268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:52.119189 16268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:52.119408 16268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:52.120061 16268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:52.120894 16268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:52.120970 16268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:52.121039 16268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:52.121085 16268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:52.131405 16268 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:37383
I20260620 19:07:52.131649 16399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:37383 every 8 connection(s)
I20260620 19:07:52.132462 16268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:07:52.140417 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 16268
Jun 20 19:07:52 dist-test-slave-6wrf krb5kdc[15357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982472, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:07:52.149811 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.135204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:34863 (local address 127.1.154.190:43519)
0620 19:07:52.135347 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.135353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.136461 (+ 1108us) server_negotiation.cc:407] Connection header received
0620 19:07:52.138350 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.138353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.138406 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.138527 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:52.140203 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.140967 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.142584 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.142868 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.142945 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.145934 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:52.145939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:52.145943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:52.145973 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:52.147560 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.148126 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.148129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.148131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.148210 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.148565 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.148568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.148570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.148816 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:52.148922 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.149454 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.149590 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:07:52.150820 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.135649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:34863)
0620 19:07:52.136245 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:52.136306 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:52.137967 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:52.138753 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:52.138761 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:52.139463 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:52.140001 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.140020 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.141163 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.141168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.142466 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.142477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.143019 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.143023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.143218 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.143920 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:52.143941 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:52.145784 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:52.147692 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:52.147698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:52.147715 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:52.148013 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:52.148302 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:52.148306 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:52.148308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:52.148459 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:52.148937 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:52.148945 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:52.149209 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 19:07:52.149396 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 19:07:52.149768 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":114,"threads_started":1}
I20260620 19:07:52.151336 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.141159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.1.154.190:43519)
0620 19:07:52.141411 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.141415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.141564 (+ 149us) server_negotiation.cc:407] Connection header received
0620 19:07:52.141756 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.141759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.141809 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.141902 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:52.142708 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.143419 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.145056 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.145388 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.145493 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.146978 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:52.146981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:52.146984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:52.147027 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:52.149110 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.149788 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.149793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.149795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.149936 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.150323 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.150327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.150329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.150537 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:52.150727 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.151014 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.151156 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":106,"threads_started":1}
I20260620 19:07:52.152199 16400 heartbeater.cc:347] Connected to a master server at 127.1.154.190:43519
I20260620 19:07:52.152490 16400 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:52.153050 16400 heartbeater.cc:511] Master 127.1.154.190:43519 requested a full tablet report, sending...
I20260620 19:07:52.154304 15936 ts_manager.cc:194] Registered new tserver with Master: 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383)
I20260620 19:07:52.155153 15936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:34863
I20260620 19:07:52.165452 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:07:52.170706 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.166415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43418 (local address 127.1.154.190:43519)
0620 19:07:52.166592 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.166597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.166612 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:07:52.166818 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.166821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.166941 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.167058 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:07:52.167315 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.167835 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.168741 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.168981 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.169103 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.169420 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:07:52.170215 (+ 795us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:07:52.170420 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.170457 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.170522 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 19:07:52.171460 15936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:07:52.174821 15936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:07:52.184365 16059 tablet_service.cc:1467] Processing CreateTablet for tablet ddf5033eb9bd4425934d64229feb3e6a (DEFAULT_TABLE table=test-table [id=ff792436dee94703adc57f9276ec0c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:52.184365 16198 tablet_service.cc:1467] Processing CreateTablet for tablet ddf5033eb9bd4425934d64229feb3e6a (DEFAULT_TABLE table=test-table [id=ff792436dee94703adc57f9276ec0c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:52.185015 16198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf5033eb9bd4425934d64229feb3e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:52.185015 16059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf5033eb9bd4425934d64229feb3e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:52.188884 16156 tablet_bootstrap.cc:492] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188: Bootstrap starting.
I20260620 19:07:52.188884 16017 tablet_bootstrap.cc:492] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc: Bootstrap starting.
I20260620 19:07:52.189941 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.183521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:37383 (local address 127.0.0.1:41950)
0620 19:07:52.183707 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:52.183726 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:52.183821 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:52.184772 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:52.184775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:52.184799 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:52.184889 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.184894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.186954 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.186958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.188162 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.188174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.189429 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.189434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.189526 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.189570 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 19:07:52.189704 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 19:07:52.189793 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 19:07:52.189962 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.183970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.1.154.131:37383)
0620 19:07:52.184279 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.184291 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.184321 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:07:52.184387 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.184391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.184593 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.185012 (+ 419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:52.185174 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.186634 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.188387 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.189288 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.189432 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.189597 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.189725 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.189822 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":131,"threads_started":1}
I20260620 19:07:52.190032 16156 tablet_bootstrap.cc:654] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:52.190058 16017 tablet_bootstrap.cc:654] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:52.192456 16156 tablet_bootstrap.cc:492] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188: No bootstrap required, opened a new log
I20260620 19:07:52.192570 16156 ts_tablet_manager.cc:1397] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 19:07:52.192725 16017 tablet_bootstrap.cc:492] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc: No bootstrap required, opened a new log
I20260620 19:07:52.192833 16017 ts_tablet_manager.cc:1397] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 19:07:52.193136 16156 raft_consensus.cc:348] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:07:52.193272 16156 raft_consensus.cc:374] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:52.193313 16156 raft_consensus.cc:729] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc97568fb1aa4aa89b6bb81caf384188, State: Initialized, Role: FOLLOWER
I20260620 19:07:52.193389 16017 raft_consensus.cc:348] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:07:52.193457 16156 consensus_queue.cc:260] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:07:52.193553 16017 raft_consensus.cc:374] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:52.193660 16017 raft_consensus.cc:729] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6ce1bdaa824703b94d50083bfffbcc, State: Initialized, Role: FOLLOWER
I20260620 19:07:52.193843 16017 consensus_queue.cc:260] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:07:52.193922 16156 ts_tablet_manager.cc:1428] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:07:52.194099 16017 ts_tablet_manager.cc:1428] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:07:52.195436 16334 tablet_service.cc:1467] Processing CreateTablet for tablet ddf5033eb9bd4425934d64229feb3e6a (DEFAULT_TABLE table=test-table [id=ff792436dee94703adc57f9276ec0c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:52.196842 16334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf5033eb9bd4425934d64229feb3e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:52.203960 16412 tablet_bootstrap.cc:492] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959: Bootstrap starting.
W20260620 19:07:52.204010 16126 tablet.cc:2369] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:07:52.206493 16412 tablet_bootstrap.cc:654] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:52.207394 16412 log.cc:824] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:52.209959 16412 tablet_bootstrap.cc:492] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959: No bootstrap required, opened a new log
I20260620 19:07:52.210139 16412 ts_tablet_manager.cc:1397] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 19:07:52.213191 16412 raft_consensus.cc:348] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:07:52.213459 16412 raft_consensus.cc:374] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:52.213493 16412 raft_consensus.cc:729] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5f44a58dd643bcb0f5666717572959, State: Initialized, Role: FOLLOWER
I20260620 19:07:52.213949 16412 consensus_queue.cc:260] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:07:52.214680 16400 heartbeater.cc:503] Master 127.1.154.190:43519 was elected leader, sending a full tablet report...
I20260620 19:07:52.215083 16412 ts_tablet_manager.cc:1428] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260620 19:07:52.369218 16131 raft_consensus.cc:482] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:52.369712 16131 raft_consensus.cc:504] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:07:52.371309 16131 leader_election.cc:288] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383), cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689)
I20260620 19:07:52.376394 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.371775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:37383 (local address 127.1.154.129:40917)
0620 19:07:52.372156 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:52.372170 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:52.372318 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:52.372610 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:52.372613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:52.372642 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:52.372799 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.372804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.373642 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.373646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.374818 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.374847 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.375781 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.375785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.375897 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.375974 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 19:07:52.376064 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 19:07:52.376186 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":7,"thread_start_us":102,"threads_started":1}
I20260620 19:07:52.376459 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.371923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:40917 (local address 127.1.154.131:37383)
0620 19:07:52.372079 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.372084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.372283 (+ 199us) server_negotiation.cc:407] Connection header received
0620 19:07:52.372458 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.372461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.372506 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.372599 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:52.372924 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.373485 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.375124 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.375612 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.375781 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.375842 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.376118 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.376227 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 19:07:52.377096 16354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf5033eb9bd4425934d64229feb3e6a" candidate_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5f44a58dd643bcb0f5666717572959" is_pre_election: true
I20260620 19:07:52.377274 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.371704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:35689 (local address 127.1.154.129:57047)
0620 19:07:52.372051 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:52.372068 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:52.372228 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:52.372575 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:52.372578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:52.372594 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:52.372745 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.372751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.373638 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.373642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.374866 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.374885 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.375924 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.375927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.376900 (+ 973us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.376931 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 19:07:52.377028 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 19:07:52.377109 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":121,"spinlock_wait_cycles":13568,"thread_start_us":111,"threads_started":1}
I20260620 19:07:52.377291 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.371833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:57047 (local address 127.1.154.130:35689)
0620 19:07:52.372106 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.372112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.372139 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:07:52.372386 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.372391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.372457 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.372563 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:52.372896 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.373489 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.375163 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.375749 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.375901 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.376117 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.377050 (+ 933us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.377155 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260620 19:07:52.377449 16354 raft_consensus.cc:2454] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff6ce1bdaa824703b94d50083bfffbcc in term 0.
I20260620 19:07:52.378005 16013 leader_election.cc:302] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e5f44a58dd643bcb0f5666717572959, ff6ce1bdaa824703b94d50083bfffbcc; no voters:
I20260620 19:07:52.377991 16202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf5033eb9bd4425934d64229feb3e6a" candidate_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc97568fb1aa4aa89b6bb81caf384188" is_pre_election: true
I20260620 19:07:52.378258 16202 raft_consensus.cc:2454] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff6ce1bdaa824703b94d50083bfffbcc in term 0.
I20260620 19:07:52.378305 16131 raft_consensus.cc:2781] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:07:52.378405 16131 raft_consensus.cc:482] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:07:52.378464 16131 raft_consensus.cc:3037] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:52.379519 16131 raft_consensus.cc:504] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:07:52.379949 16131 leader_election.cc:288] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 1 election: Requested vote from peers 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383), cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689)
I20260620 19:07:52.380280 16354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf5033eb9bd4425934d64229feb3e6a" candidate_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5f44a58dd643bcb0f5666717572959"
I20260620 19:07:52.380419 16354 raft_consensus.cc:3037] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:52.380450 16202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf5033eb9bd4425934d64229feb3e6a" candidate_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc97568fb1aa4aa89b6bb81caf384188"
I20260620 19:07:52.380609 16202 raft_consensus.cc:3037] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:52.381524 16354 raft_consensus.cc:2454] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6ce1bdaa824703b94d50083bfffbcc in term 1.
I20260620 19:07:52.381718 16202 raft_consensus.cc:2454] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6ce1bdaa824703b94d50083bfffbcc in term 1.
I20260620 19:07:52.381873 16013 leader_election.cc:302] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e5f44a58dd643bcb0f5666717572959, ff6ce1bdaa824703b94d50083bfffbcc; no voters:
I20260620 19:07:52.382071 16131 raft_consensus.cc:2781] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:52.382283 16131 raft_consensus.cc:686] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 LEADER]: Becoming Leader. State: Replica: ff6ce1bdaa824703b94d50083bfffbcc, State: Running, Role: LEADER
I20260620 19:07:52.382550 16131 consensus_queue.cc:237] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
W20260620 19:07:52.383524 16401 tablet.cc:2369] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:07:52.385880 15936 catalog_manager.cc:5495] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc reported cstate change: term changed from 0 to 1, leader changed from <none> to ff6ce1bdaa824703b94d50083bfffbcc (127.1.154.129). New cstate: current_term: 1 leader_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } health_report { overall_health: UNKNOWN } } }
W20260620 19:07:52.421269 16266 tablet.cc:2369] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:07:52.433197 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.429147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.1.154.129:40511)
0620 19:07:52.429423 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.429429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.429441 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:07:52.429515 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.429517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.429570 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.429691 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:07:52.429971 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.430507 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.431652 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.431901 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.432000 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.432338 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:07:52.432782 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:07:52.432929 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.432972 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.433031 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":117,"threads_started":1}
I20260620 19:07:52.437076 16202 raft_consensus.cc:1261] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188 [term 1 FOLLOWER]: Refusing update from remote peer ff6ce1bdaa824703b94d50083bfffbcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:52.437088 16354 raft_consensus.cc:1261] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959 [term 1 FOLLOWER]: Refusing update from remote peer ff6ce1bdaa824703b94d50083bfffbcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:52.437801 16131 consensus_queue.cc:1035] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:52.438020 16417 consensus_queue.cc:1035] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:52.454808 15935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:52.455008 15935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:52.459715 15935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: Sending DeleteTablet for 3 replicas of tablet ddf5033eb9bd4425934d64229feb3e6a
I20260620 19:07:52.460687 16059 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf5033eb9bd4425934d64229feb3e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:52 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:60088
I20260620 19:07:52.460785 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 15975
I20260620 19:07:52.461107 16334 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf5033eb9bd4425934d64229feb3e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:52 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:41950
I20260620 19:07:52.463109 16197 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf5033eb9bd4425934d64229feb3e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:52 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:60548
I20260620 19:07:52.463102 16198 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf5033eb9bd4425934d64229feb3e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:52 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:60548
I20260620 19:07:52.463722 16430 tablet_replica.cc:330] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959: stopping tablet replica
I20260620 19:07:52.464174 16430 raft_consensus.cc:2229] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:52.464569 16430 raft_consensus.cc:2258] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:52.466825 16271 tablet_replica.cc:330] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188: stopping tablet replica
I20260620 19:07:52.466989 16271 raft_consensus.cc:2229] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:52.467100 16430 ts_tablet_manager.cc:1905] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:52.467146 16271 raft_consensus.cc:2258] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:52.467993 16271 ts_tablet_manager.cc:1905] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:52.470453 16430 ts_tablet_manager.cc:1918] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:07:52.470592 16430 log.cc:1196] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/ddf5033eb9bd4425934d64229feb3e6a
I20260620 19:07:52.470983 16430 ts_tablet_manager.cc:1939] T ddf5033eb9bd4425934d64229feb3e6a P 6e5f44a58dd643bcb0f5666717572959: Deleting consensus metadata
I20260620 19:07:52.471431 16271 ts_tablet_manager.cc:1918] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:07:52.471520 16271 log.cc:1196] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/ddf5033eb9bd4425934d64229feb3e6a
I20260620 19:07:52.471813 16271 ts_tablet_manager.cc:1939] T ddf5033eb9bd4425934d64229feb3e6a P cc97568fb1aa4aa89b6bb81caf384188: Deleting consensus metadata
I20260620 19:07:52.472443 15923 catalog_manager.cc:4841] TS 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383): tablet ddf5033eb9bd4425934d64229feb3e6a (table test-table [id=ff792436dee94703adc57f9276ec0c22]) successfully deleted
I20260620 19:07:52.472569 15924 catalog_manager.cc:4841] TS cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689): tablet ddf5033eb9bd4425934d64229feb3e6a (table test-table [id=ff792436dee94703adc57f9276ec0c22]) successfully deleted
W20260620 19:07:52.473019 15924 catalog_manager.cc:4808] TS cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689): delete failed for tablet ddf5033eb9bd4425934d64229feb3e6a because the tablet was not found. No further retry: Not found: Tablet not found: ddf5033eb9bd4425934d64229feb3e6a
W20260620 19:07:52.473295 15921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:40511 (error 108)
I20260620 19:07:52.474109 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 16130
I20260620 19:07:52.475914 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.474080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:40511 (local address 127.0.0.1:60116)
0620 19:07:52.475742 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:52.475822 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1565}
W20260620 19:07:52.476786 15921 catalog_manager.cc:4584] TS ff6ce1bdaa824703b94d50083bfffbcc (127.1.154.129:40511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ddf5033eb9bd4425934d64229feb3e6a: Network error: Client connection negotiation failed: client connection to 127.1.154.129:40511: connect: Connection refused (error 111)
I20260620 19:07:52.482455 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 16268
I20260620 19:07:52.488464 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 15904
I20260620 19:07:52.495020 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43519
--webserver_interface=127.1.154.190
--webserver_port=38325
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:43519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:52.617873 16432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:52.618212 16432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:52.618314 16432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:52.622557 16432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:07:52.622649 16432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:52.622686 16432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:52.622745 16432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:07:52.622783 16432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:07:52.628350 16432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:43519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=38325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:52.629544 16432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:52.630611 16432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:52.641052 16440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:52.641115 16438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:52.641295 16432 server_base.cc:1034] running on GCE node
W20260620 19:07:52.641167 16437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:52.641968 16432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:52.643107 16432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:52.644312 16432 hybrid_clock.cc:648] HybridClock initialized: now 1781982472644284 us; error 52 us; skew 500 ppm
Jun 20 19:07:52 dist-test-slave-6wrf krb5kdc[15357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982472, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:52.647393 16432 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:07:52.648365 16432 webserver.cc:460] Webserver started at http://127.1.154.190:38325/ using document root <none> and password file <none>
I20260620 19:07:52.648962 16432 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:52.649055 16432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:52.653192 16432 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:07:52.655840 16447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:52.657763 16432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 19:07:52.657943 16432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "6eb8583552164d88a308122bd7b04b2e"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "7a7726dc58227f7c9308541bfb26464e"
server_key_iv: "6a340960fb1863333ee57a92a5873dd4"
server_key_version: "encryptionkey@0"
I20260620 19:07:52.658561 16432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:52.672930 16432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:52.676573 16432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:52.676807 16432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:52.685587 16432 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:43519
I20260620 19:07:52.685659 16499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:43519 every 8 connection(s)
I20260620 19:07:52.687458 16432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:07:52.692607 16500 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:07:52.693188 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 16432
I20260620 19:07:52.693682 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:40511
--local_ip_for_outbound_sockets=127.1.154.129
--tserver_master_addrs=127.1.154.190:43519
--webserver_port=40851
--webserver_interface=127.1.154.129
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:07:52.694845 16500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: Bootstrap starting.
I20260620 19:07:52.719617 16500 log.cc:824] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:52.733898 16500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:07:52.734375 16500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e: Bootstrap complete.
I20260620 19:07:52.739970 16500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } }
I20260620 19:07:52.740514 16500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eb8583552164d88a308122bd7b04b2e, State: Initialized, Role: FOLLOWER
I20260620 19:07:52.741099 16500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } }
I20260620 19:07:52.741263 16500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:07:52.741324 16500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:07:52.741410 16500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:07:52.744045 16500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } }
I20260620 19:07:52.744423 16500 leader_election.cc:302] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eb8583552164d88a308122bd7b04b2e; no voters:
I20260620 19:07:52.744781 16500 leader_election.cc:288] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 19:07:52.745052 16504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:07:52.745337 16504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [term 3 LEADER]: Becoming Leader. State: Replica: 6eb8583552164d88a308122bd7b04b2e, State: Running, Role: LEADER
I20260620 19:07:52.745692 16504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } }
I20260620 19:07:52.746202 16500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:07:52.747085 16505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6eb8583552164d88a308122bd7b04b2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } } }
I20260620 19:07:52.747227 16505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:52.747550 16506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eb8583552164d88a308122bd7b04b2e. Latest consensus state: current_term: 3 leader_uuid: "6eb8583552164d88a308122bd7b04b2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8583552164d88a308122bd7b04b2e" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43519 } } }
I20260620 19:07:52.747653 16506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb8583552164d88a308122bd7b04b2e [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:52.748485 16513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:07:52.754897 16513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:07:52.755846 16513 catalog_manager.cc:1259] Loaded cluster ID: 231a348e3ef24c5782d7a2003c69bf3c
I20260620 19:07:52.755968 16513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 19:07:52.757833 16513 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd5c1e679ad google::LogMessage::Fail() at ??:0
@ 0x7fd5c1e6a88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd5c1e674de google::LogMessage::Flush() at ??:0
@ 0x7fd5c1e68a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd5cc7a6fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd5cc7a7d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd5cc7a4579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fd5cc7e8640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd5c40b93d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd5c40b9c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd5c40bb3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd5c40aa512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd5c56ee6db start_thread at ??:0
@ 0x7fd5c096561f clone at ??:0
W20260620 19:07:52.855286 16502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:52.855631 16502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:52.855713 16502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:52.860612 16502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:52.860715 16502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:52.860808 16502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:07:52.866818 16502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:40511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=40851
--tserver_master_addrs=127.1.154.190:43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:52.867991 16502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:52.869055 16502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:52.879190 16544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:52.879215 16542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:52.879330 16541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:52.879370 16502 server_base.cc:1034] running on GCE node
I20260620 19:07:52.879849 16502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:52.880445 16502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:52.881675 16502 hybrid_clock.cc:648] HybridClock initialized: now 1781982472881659 us; error 44 us; skew 500 ppm
Jun 20 19:07:52 dist-test-slave-6wrf krb5kdc[15357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982472, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:52.884752 16502 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:07:52.885751 16502 webserver.cc:460] Webserver started at http://127.1.154.129:40851/ using document root <none> and password file <none>
I20260620 19:07:52.886361 16502 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:52.886430 16502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:52.890920 16502 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 19:07:52.894420 16551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:52.896075 16502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:07:52.896188 16502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "ff6ce1bdaa824703b94d50083bfffbcc"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "997bef6c605b0668d686e6f9689e619d"
server_key_iv: "116556c4beb607782c9e54a5ca48d5de"
server_key_version: "encryptionkey@0"
I20260620 19:07:52.896802 16502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:52.909963 16502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:52.914722 16502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:52.915033 16502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:52.915894 16502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:52.917497 16558 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:07:52.923353 16502 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:07:52.923522 16502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 19:07:52.923635 16502 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:07:52.926331 16502 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:07:52.926429 16502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260620 19:07:52.926774 16558 tablet_bootstrap.cc:492] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc: Bootstrap starting.
I20260620 19:07:52.940371 16502 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:40511
I20260620 19:07:52.940685 16665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:40511 every 8 connection(s)
I20260620 19:07:52.941572 16502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:07:52.943032 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 16502
I20260620 19:07:52.943666 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:35689
--local_ip_for_outbound_sockets=127.1.154.130
--tserver_master_addrs=127.1.154.190:43519
--webserver_port=43477
--webserver_interface=127.1.154.130
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:07:52.953387 16558 log.cc:824] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:52.963203 16558 tablet_bootstrap.cc:492] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:07:52.963668 16558 tablet_bootstrap.cc:492] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc: Bootstrap complete.
I20260620 19:07:52.965263 16558 ts_tablet_manager.cc:1397] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc: Time spent bootstrapping tablet: real 0.039s user 0.026s sys 0.007s
I20260620 19:07:52.969028 16558 raft_consensus.cc:348] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:07:52.969688 16558 raft_consensus.cc:729] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6ce1bdaa824703b94d50083bfffbcc, State: Initialized, Role: FOLLOWER
I20260620 19:07:52.970304 16558 consensus_queue.cc:260] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:07:52.971468 16558 ts_tablet_manager.cc:1428] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc: Time spent starting tablet: real 0.006s user 0.009s sys 0.000s
W20260620 19:07:53.012509 16668 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:07:52.943824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:36799)
0620 19:07:52.945107 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:52.945165 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:52.946421 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:53.011745 (+ 65324us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1046,"thread_start_us":154,"threads_started":1}
W20260620 19:07:53.013103 16666 heartbeater.cc:644] Failed to heartbeat to 127.1.154.190:43519 (0 consecutive failures): Network error: Failed to ping master at 127.1.154.190:43519: Client connection negotiation failed: client connection to 127.1.154.190:43519: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:07:53.013861 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:53.013574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:43843)
0620 19:07:53.013716 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:53.013804 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 19:07:53.014858 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:53.014571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:50825)
0620 19:07:53.014725 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:53.014794 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 19:07:53.015254 16666 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:07:53.074440 16669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:53.074705 16669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:53.074751 16669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:53.078743 16669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:53.078812 16669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:53.078872 16669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:07:53.084297 16669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:35689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=43477
--tserver_master_addrs=127.1.154.190:43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:53.085418 16669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:53.086515 16669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:53.096647 16679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:53.096647 16676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:53.096683 16677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:53.097370 16669 server_base.cc:1034] running on GCE node
I20260620 19:07:53.098116 16669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:53.099176 16669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:53.100559 16669 hybrid_clock.cc:648] HybridClock initialized: now 1781982473100448 us; error 125 us; skew 500 ppm
Jun 20 19:07:53 dist-test-slave-6wrf krb5kdc[15357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982473, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:53.104513 16669 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:07:53.105577 16669 webserver.cc:460] Webserver started at http://127.1.154.130:43477/ using document root <none> and password file <none>
I20260620 19:07:53.106232 16669 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:53.106324 16669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:53.110384 16669 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 19:07:53.112881 16686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:53.114786 16669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:07:53.114964 16669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "cc97568fb1aa4aa89b6bb81caf384188"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "6464c1a63600962b5c7eb32a5dd3f91b"
server_key_iv: "7c1b92783a43d205940ea717f43bc0e5"
server_key_version: "encryptionkey@0"
I20260620 19:07:53.115851 16669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:53.129062 16669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:53.133038 16669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:53.133211 16669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:53.133841 16669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:53.134631 16669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:53.134699 16669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:53.134742 16669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:53.134757 16669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:53.147610 16669 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:35689
I20260620 19:07:53.147691 16799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:35689 every 8 connection(s)
I20260620 19:07:53.148808 16669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:07:53.153012 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 16669
I20260620 19:07:53.153081 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:53.151390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:47483)
0620 19:07:53.152126 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:53.152241 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":122,"threads_started":1}
W20260620 19:07:53.153482 16800 heartbeater.cc:644] Failed to heartbeat to 127.1.154.190:43519 (0 consecutive failures): Network error: Failed to ping master at 127.1.154.190:43519: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
I20260620 19:07:53.153561 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:37383
--local_ip_for_outbound_sockets=127.1.154.131
--tserver_master_addrs=127.1.154.190:43519
--webserver_port=34155
--webserver_interface=127.1.154.131
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:07:53.154572 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:53.154194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:47961)
0620 19:07:53.154402 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:53.154479 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 19:07:53.155328 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:53.155114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:33015)
0620 19:07:53.155218 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:53.155265 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 19:07:53.155515 16800 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:07:53.271978 16803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:53.272281 16803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:53.272327 16803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:53.276567 16803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:53.276631 16803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:53.276695 16803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:07:53.282512 16803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:37383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=34155
--tserver_master_addrs=127.1.154.190:43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:07:53.283618 16803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:53.284618 16803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:53.295321 16811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:53.295328 16809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:53.295326 16808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:53.295583 16803 server_base.cc:1034] running on GCE node
I20260620 19:07:53.296105 16803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:53.296769 16803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:53.297974 16803 hybrid_clock.cc:648] HybridClock initialized: now 1781982473297916 us; error 71 us; skew 500 ppm
Jun 20 19:07:53 dist-test-slave-6wrf krb5kdc[15357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982473, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:53.301443 16803 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:07:53.302950 16803 webserver.cc:460] Webserver started at http://127.1.154.131:34155/ using document root <none> and password file <none>
I20260620 19:07:53.303834 16803 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:53.303947 16803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:53.308557 16803 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:07:53.310824 16818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:53.312399 16803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:07:53.312561 16803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "6e5f44a58dd643bcb0f5666717572959"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-6wrf"
server_key: "18c37a2a989321177cd9a5a3d001c87b"
server_key_iv: "f3471c0e966bab9d1e6c663f9d1906e0"
server_key_version: "encryptionkey@0"
I20260620 19:07:53.313129 16803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:53.343797 16803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:53.347631 16803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:53.347875 16803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:53.348490 16803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:53.349311 16803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:53.349382 16803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:53.349454 16803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:53.349498 16803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:53.359020 16803 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:37383
I20260620 19:07:53.359057 16931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:37383 every 8 connection(s)
I20260620 19:07:53.360525 16803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:07:53.361748 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 16803
I20260620 19:07:53.364650 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:53.363081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:41905)
0620 19:07:53.363534 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:53.363971 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":108,"threads_started":1}
W20260620 19:07:53.365221 16932 heartbeater.cc:644] Failed to heartbeat to 127.1.154.190:43519 (0 consecutive failures): Network error: Failed to ping master at 127.1.154.190:43519: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
I20260620 19:07:53.365914 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:53.365713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:50949)
0620 19:07:53.365805 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:53.365858 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 19:07:53.366609 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:53.366386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:57679)
0620 19:07:53.366514 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:53.366560 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 19:07:53.366864 16932 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:07:54.016811 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.016181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:52269)
0620 19:07:54.016565 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.016648 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
I20260620 19:07:54.157457 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.156716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:57209)
0620 19:07:54.157194 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.157281 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":151,"threads_started":1}
I20260620 19:07:54.368314 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.367674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:45801)
0620 19:07:54.368057 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.368147 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":102,"threads_started":1}
I20260620 19:07:54.554651 16939 raft_consensus.cc:482] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:54.554934 16939 raft_consensus.cc:504] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:07:54.556604 16939 leader_election.cc:288] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383), cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689)
Jun 20 19:07:54 dist-test-slave-6wrf krb5kdc[15357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982472, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.131@KRBTEST.COM
Jun 20 19:07:54 dist-test-slave-6wrf krb5kdc[15357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982472, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.130@KRBTEST.COM
I20260620 19:07:54.576180 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.556939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:35689 (local address 127.1.154.129:35691)
0620 19:07:54.557928 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.557944 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.558065 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.560023 (+ 1958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.560031 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.560642 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:54.561176 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.561203 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.562825 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.562829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.564076 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.564089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.564962 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.564970 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.565224 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.566114 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:54.566152 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:54.568675 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:54.571899 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:54.571908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:54.571929 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:54.572589 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:54.573236 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:54.573241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:54.573243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:54.573591 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:54.574718 (+ 1127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:54.574726 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:54.575224 (+ 498us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.575485 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.575975 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":498,"spinlock_wait_cycles":66432,"thread_start_us":158,"threads_started":1}
I20260620 19:07:54.576181 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.556939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:37383 (local address 127.1.154.129:36053)
0620 19:07:54.557533 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.557553 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.558039 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.560023 (+ 1984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.560031 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.560806 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:54.561178 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.561207 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.563212 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.563216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.564066 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.564077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.565045 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.565049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.565239 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.566124 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:54.566141 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:54.568105 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:54.571899 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:54.571908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:54.571929 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:54.572589 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:54.573236 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:54.573241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:54.573243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:54.573591 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:54.574718 (+ 1127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:54.574726 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:54.575224 (+ 498us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.575485 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.575918 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":178,"spinlock_wait_cycles":22272,"thread_start_us":114,"threads_started":1}
I20260620 19:07:54.576699 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.557460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:35691 (local address 127.1.154.130:35689)
0620 19:07:54.558022 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.558050 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.558104 (+ 54us) server_negotiation.cc:407] Connection header received
0620 19:07:54.559037 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.559045 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.559437 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.559931 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:54.561410 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.562591 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.564275 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.564798 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.564937 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.569041 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:54.569050 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:54.569069 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:54.569123 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:54.571662 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.572852 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.572858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.572862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.573053 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.573794 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.573801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.573805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.574531 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:54.574710 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.575770 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.576155 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":159,"threads_started":1}
I20260620 19:07:54.576785 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.557403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:36053 (local address 127.1.154.131:37383)
0620 19:07:54.558011 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.558044 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.558105 (+ 61us) server_negotiation.cc:407] Connection header received
0620 19:07:54.558963 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.558971 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.559437 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.559850 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:54.561487 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.563017 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.564231 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.564845 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.565086 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.568614 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:54.568623 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:54.568647 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:54.568695 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:54.571654 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.572818 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.572827 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.572834 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.573054 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.573748 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.573752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.573753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.574531 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:54.574710 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.575733 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.576155 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":101,"threads_started":1}
W20260620 19:07:54.580050 16555 leader_election.cc:341] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689): Not found: Tablet not found: ddf5033eb9bd4425934d64229feb3e6a
W20260620 19:07:54.580288 16554 leader_election.cc:341] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383): Not found: Tablet not found: ddf5033eb9bd4425934d64229feb3e6a
I20260620 19:07:54.580402 16554 leader_election.cc:302] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff6ce1bdaa824703b94d50083bfffbcc; no voters: 6e5f44a58dd643bcb0f5666717572959, cc97568fb1aa4aa89b6bb81caf384188
I20260620 19:07:54.580786 16939 raft_consensus.cc:2726] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:07:55.018371 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:55.017967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:33519)
0620 19:07:55.018179 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:55.018277 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 19:07:55.159251 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:55.158592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:60321)
0620 19:07:55.159042 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:55.159132 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":130,"threads_started":1}
I20260620 19:07:55.370479 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:55.369737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:33063)
0620 19:07:55.370222 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:55.370341 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":89,"threads_started":1}
I20260620 19:07:56.020215 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.019561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:43633)
0620 19:07:56.020018 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.020102 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":145,"threads_started":1}
I20260620 19:07:56.161127 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.160464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:37437)
0620 19:07:56.160915 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.160995 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":261,"threads_started":1}
I20260620 19:07:56.372290 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.371664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:32935)
0620 19:07:56.372052 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.372139 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":109,"threads_started":1}
I20260620 19:07:56.533020 16949 raft_consensus.cc:482] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:56.533438 16949 raft_consensus.cc:504] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:07:56.534462 16949 leader_election.cc:288] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383), cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689)
W20260620 19:07:56.535415 16554 leader_election.cc:341] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383): Not found: Tablet not found: ddf5033eb9bd4425934d64229feb3e6a
W20260620 19:07:56.535641 16555 leader_election.cc:341] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689): Not found: Tablet not found: ddf5033eb9bd4425934d64229feb3e6a
I20260620 19:07:56.535887 16555 leader_election.cc:302] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff6ce1bdaa824703b94d50083bfffbcc; no voters: 6e5f44a58dd643bcb0f5666717572959, cc97568fb1aa4aa89b6bb81caf384188
I20260620 19:07:56.536093 16949 raft_consensus.cc:2726] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:07:57.022245 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:57.021565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:55253)
0620 19:07:57.022053 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:57.022139 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":141,"threads_started":1}
I20260620 19:07:57.162904 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:57.162181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:50057)
0620 19:07:57.162657 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:57.162754 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":94,"threads_started":1}
I20260620 19:07:57.374015 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:57.373364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:39837)
0620 19:07:57.373822 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:57.373906 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":140,"threads_started":1}
I20260620 19:07:58.024392 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:58.023611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:47183)
0620 19:07:58.024161 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:58.024273 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":109,"threads_started":1}
I20260620 19:07:58.164585 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:58.164113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:49863)
0620 19:07:58.164406 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:58.164469 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":96,"threads_started":1}
I20260620 19:07:58.375788 16955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:58.375285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:48045)
0620 19:07:58.375611 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:58.375669 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":129,"threads_started":1}
I20260620 19:07:59.026325 16956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:59.025726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:40225)
0620 19:07:59.026133 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:59.026222 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":116,"threads_started":1}
I20260620 19:07:59.166193 16957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:59.165417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:44611)
0620 19:07:59.165943 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:59.166045 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":81,"threads_started":1}
I20260620 19:07:59.195497 16958 raft_consensus.cc:482] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:59.195685 16958 raft_consensus.cc:504] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:07:59.196367 16958 leader_election.cc:288] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383), cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689)
W20260620 19:07:59.197453 16554 leader_election.cc:341] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383): Not found: Tablet not found: ddf5033eb9bd4425934d64229feb3e6a
W20260620 19:07:59.197765 16555 leader_election.cc:341] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689): Not found: Tablet not found: ddf5033eb9bd4425934d64229feb3e6a
I20260620 19:07:59.197872 16555 leader_election.cc:302] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff6ce1bdaa824703b94d50083bfffbcc; no voters: 6e5f44a58dd643bcb0f5666717572959, cc97568fb1aa4aa89b6bb81caf384188
I20260620 19:07:59.198040 16958 raft_consensus.cc:2726] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:07:59.377537 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:59.377054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:33267)
0620 19:07:59.377368 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:59.377429 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":118,"threads_started":1}
I20260620 19:08:00.028129 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:00.027460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:53979)
0620 19:08:00.027881 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:00.027966 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":145,"threads_started":1}
I20260620 19:08:00.167670 16961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:00.167162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:44743)
0620 19:08:00.167513 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:00.167574 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":90,"threads_started":1}
I20260620 19:08:00.378934 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:00.378416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:46733)
0620 19:08:00.378777 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:00.378838 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":93,"threads_started":1}
I20260620 19:08:01.010248 16963 raft_consensus.cc:482] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:01.010450 16963 raft_consensus.cc:504] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:08:01.011389 16963 leader_election.cc:288] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383), cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689)
W20260620 19:08:01.012259 16554 leader_election.cc:341] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383): Not found: Tablet not found: ddf5033eb9bd4425934d64229feb3e6a
W20260620 19:08:01.012346 16555 leader_election.cc:341] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689): Not found: Tablet not found: ddf5033eb9bd4425934d64229feb3e6a
I20260620 19:08:01.012388 16554 leader_election.cc:302] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff6ce1bdaa824703b94d50083bfffbcc; no voters: 6e5f44a58dd643bcb0f5666717572959, cc97568fb1aa4aa89b6bb81caf384188
I20260620 19:08:01.012518 16963 raft_consensus.cc:2726] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:08:01.029575 16964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:01.029143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:41579)
0620 19:08:01.029421 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:01.029476 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":100,"threads_started":1}
I20260620 19:08:01.169566 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:01.168845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:36759)
0620 19:08:01.169301 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:01.169379 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":113,"threads_started":1}
I20260620 19:08:01.380764 16966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:01.380106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:35991)
0620 19:08:01.380567 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:01.380657 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":162,"threads_started":1}
I20260620 19:08:02.031139 16967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:02.030711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:49973)
0620 19:08:02.030972 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:02.031030 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
I20260620 19:08:02.171545 16968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:02.170635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:45153)
0620 19:08:02.171112 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:02.171222 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":93,"threads_started":1}
I20260620 19:08:02.382609 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:02.381760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:34343)
0620 19:08:02.382199 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:02.382333 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":86,"threads_started":1}
I20260620 19:08:03.033373 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:03.032796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:40341)
0620 19:08:03.033189 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:03.033271 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":114,"threads_started":1}
I20260620 19:08:03.173909 16971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:03.173175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:35115)
0620 19:08:03.173608 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:03.173780 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":123,"threads_started":1}
I20260620 19:08:03.384418 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:03.383899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:34385)
0620 19:08:03.384234 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:03.384310 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":94,"threads_started":1}
I20260620 19:08:04.035362 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:04.034717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:41759)
0620 19:08:04.035142 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:04.035244 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":111,"threads_started":1}
I20260620 19:08:04.175496 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:04.174821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:53991)
0620 19:08:04.175259 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:04.175363 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":103,"threads_started":1}
I20260620 19:08:04.359308 16975 raft_consensus.cc:482] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:04.359472 16975 raft_consensus.cc:504] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:08:04.359926 16975 leader_election.cc:288] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383), cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689)
W20260620 19:08:04.360796 16555 leader_election.cc:341] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689): Not found: Tablet not found: ddf5033eb9bd4425934d64229feb3e6a
W20260620 19:08:04.360891 16554 leader_election.cc:341] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383): Not found: Tablet not found: ddf5033eb9bd4425934d64229feb3e6a
I20260620 19:08:04.361013 16554 leader_election.cc:302] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff6ce1bdaa824703b94d50083bfffbcc; no voters: 6e5f44a58dd643bcb0f5666717572959, cc97568fb1aa4aa89b6bb81caf384188
I20260620 19:08:04.361159 16975 raft_consensus.cc:2726] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:08:04.385874 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:04.385318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:38895)
0620 19:08:04.385698 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:04.385773 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260620 19:08:05.037717 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:05.036905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:38857)
0620 19:08:05.037449 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:05.037531 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":122,"threads_started":1}
I20260620 19:08:05.177220 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:05.176545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:35389)
0620 19:08:05.176978 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:05.177074 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":98,"threads_started":1}
I20260620 19:08:05.387645 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:05.387059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:49663)
0620 19:08:05.387421 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:05.387500 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":118,"threads_started":1}
I20260620 19:08:06.039482 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:06.038853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:51515)
0620 19:08:06.039247 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:06.039343 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":98,"threads_started":1}
I20260620 19:08:06.179203 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:06.178504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:48843)
0620 19:08:06.178994 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:06.179077 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":147,"threads_started":1}
I20260620 19:08:06.389734 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:06.388905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:37845)
0620 19:08:06.389289 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:06.389515 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":106,"threads_started":1}
I20260620 19:08:06.823761 16983 raft_consensus.cc:482] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:06.824131 16983 raft_consensus.cc:504] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e5f44a58dd643bcb0f5666717572959" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 37383 } } peers { permanent_uuid: "ff6ce1bdaa824703b94d50083bfffbcc" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40511 } } peers { permanent_uuid: "cc97568fb1aa4aa89b6bb81caf384188" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 35689 } }
I20260620 19:08:06.824743 16983 leader_election.cc:288] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383), cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689)
W20260620 19:08:06.825774 16554 leader_election.cc:341] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6e5f44a58dd643bcb0f5666717572959 (127.1.154.131:37383): Not found: Tablet not found: ddf5033eb9bd4425934d64229feb3e6a
W20260620 19:08:06.825973 16555 leader_election.cc:341] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc97568fb1aa4aa89b6bb81caf384188 (127.1.154.130:35689): Not found: Tablet not found: ddf5033eb9bd4425934d64229feb3e6a
I20260620 19:08:06.826119 16555 leader_election.cc:302] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff6ce1bdaa824703b94d50083bfffbcc; no voters: 6e5f44a58dd643bcb0f5666717572959, cc97568fb1aa4aa89b6bb81caf384188
I20260620 19:08:06.826304 16983 raft_consensus.cc:2726] T ddf5033eb9bd4425934d64229feb3e6a P ff6ce1bdaa824703b94d50083bfffbcc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:08:07.041713 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:07.041068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:35721)
0620 19:08:07.041466 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:07.041533 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":116,"threads_started":1}
I20260620 19:08:07.180869 16985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:07.180280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:33975)
0620 19:08:07.180676 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:07.180760 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":134,"threads_started":1}
I20260620 19:08:07.391430 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:07.390921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.131:56277)
0620 19:08:07.391254 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:07.391319 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":89,"threads_started":1}
I20260620 19:08:08.043525 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:08.043064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.129:48157)
0620 19:08:08.043355 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:08.043414 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
I20260620 19:08:08.182358 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:08.181906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43519 (local address 127.1.154.130:54233)
0620 19:08:08.182205 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:08.182265 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":81,"threads_started":1}
I20260620 19:08:08.371925 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 16502
I20260620 19:08:08.378391 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 16669
I20260620 19:08:08.384377 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 16803
2026-06-20T19:08:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19705 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:08:08 dist-test-slave-6wrf krb5kdc[16991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:08:08 dist-test-slave-6wrf krb5kdc[16991](info): set up 2 sockets
Jun 20 19:08:08 dist-test-slave-6wrf krb5kdc[16991](info): commencing operation
krb5kdc: starting...
W20260620 19:08:09.503515 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:08:09 dist-test-slave-6wrf krb5kdc[16991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:08:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:08:09.645848 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:35761
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:35761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:09.775434 17004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:09.775804 17004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:09.776000 17004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:09.781281 17004 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:08:09.781458 17004 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:08:09.781490 17004 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:08:09.781517 17004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:09.781570 17004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:09.781663 17004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:09.781723 17004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:09.781790 17004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:09.790324 17004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:35761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:35761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:08:09.792021 17004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:09.793373 17004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:09.803169 17010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:09.803732 17009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:09.803736 17012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:09.804301 17004 server_base.cc:1034] running on GCE node
I20260620 19:08:09.804951 17004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:09.806892 17004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:09.808138 17004 hybrid_clock.cc:648] HybridClock initialized: now 1781982489808075 us; error 92 us; skew 500 ppm
Jun 20 19:08:09 dist-test-slave-6wrf krb5kdc[16991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:09.811383 17004 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:08:09.812444 17004 webserver.cc:460] Webserver started at http://127.1.154.190:35601/ using document root <none> and password file <none>
I20260620 19:08:09.813084 17004 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:09.813156 17004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:09.813400 17004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:09.815690 17004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "1fbc3766918b45ceb67294eb0ae91f3d"
format_stamp: "Formatted at 2026-06-20 19:08:09 on dist-test-slave-6wrf"
server_key: "6eee58b40b2e04583d146084f7691338"
server_key_iv: "f86725cc54471aa6e64050bc64293ba8"
server_key_version: "encryptionkey@0"
I20260620 19:08:09.816301 17004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "1fbc3766918b45ceb67294eb0ae91f3d"
format_stamp: "Formatted at 2026-06-20 19:08:09 on dist-test-slave-6wrf"
server_key: "6eee58b40b2e04583d146084f7691338"
server_key_iv: "f86725cc54471aa6e64050bc64293ba8"
server_key_version: "encryptionkey@0"
I20260620 19:08:09.820413 17004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 19:08:09.823316 17019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:09.824937 17004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:08:09.825107 17004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "1fbc3766918b45ceb67294eb0ae91f3d"
format_stamp: "Formatted at 2026-06-20 19:08:09 on dist-test-slave-6wrf"
server_key: "6eee58b40b2e04583d146084f7691338"
server_key_iv: "f86725cc54471aa6e64050bc64293ba8"
server_key_version: "encryptionkey@0"
I20260620 19:08:09.825234 17004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:09.837909 17004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:09.842170 17004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:09.842489 17004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:09.852221 17004 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:35761
I20260620 19:08:09.852618 17071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:35761 every 8 connection(s)
I20260620 19:08:09.853317 17004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:09.855077 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 17004
I20260620 19:08:09.855293 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:08:09.855896 1642 external_mini_cluster.cc:1428] Setting key 44c4729e21042e72173e4aaedd433912
I20260620 19:08:09.858573 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:09.866729 17072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Bootstrap starting.
Jun 20 19:08:09 dist-test-slave-6wrf krb5kdc[16991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:08:09.869240 17072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:09.870273 17072 log.cc:824] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:09.873749 17072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: No bootstrap required, opened a new log
I20260620 19:08:09.875991 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:09.858323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60238 (local address 127.1.154.190:35761)
0620 19:08:09.859110 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:09.859136 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:09.859189 (+ 53us) server_negotiation.cc:407] Connection header received
0620 19:08:09.860098 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:09.860121 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:09.860561 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:09.861187 (+ 626us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:09.862018 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:09.863595 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:09.864520 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:09.865261 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:09.865431 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:09.868982 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:09.868988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:09.869004 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:09.869048 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:09.871921 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:09.872529 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:09.872537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:09.872544 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:09.872731 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:09.873078 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:09.873082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:09.873084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:09.873784 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:09.874005 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:09.874373 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 19:08:09.874969 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":224,"threads_started":1}
I20260620 19:08:09.878728 17072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } }
I20260620 19:08:09.879037 17072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:09.879093 17072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fbc3766918b45ceb67294eb0ae91f3d, State: Initialized, Role: FOLLOWER
I20260620 19:08:09.879510 17072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } }
I20260620 19:08:09.879642 17072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:09.879714 17072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:09.879817 17072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:09.880857 17072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } }
I20260620 19:08:09.881245 17072 leader_election.cc:302] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fbc3766918b45ceb67294eb0ae91f3d; no voters:
I20260620 19:08:09.881635 17072 leader_election.cc:288] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:09.881728 17077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:09.881906 17077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 1 LEADER]: Becoming Leader. State: Replica: 1fbc3766918b45ceb67294eb0ae91f3d, State: Running, Role: LEADER
I20260620 19:08:09.882220 17077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } }
I20260620 19:08:09.882939 17072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:09.884511 17079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } } }
I20260620 19:08:09.884745 17078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fbc3766918b45ceb67294eb0ae91f3d. Latest consensus state: current_term: 1 leader_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } } }
I20260620 19:08:09.884943 17078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:09.885356 17079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:09.885449 17087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:09.888797 17087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:09.894659 17087 catalog_manager.cc:1347] Generated new cluster ID: fb6d796b51a7402e9ef44d36bfc79661
I20260620 19:08:09.894739 17087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:09.912863 17087 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:09.914354 17087 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:09.923980 17087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Generated new TSK 0
I20260620 19:08:09.924953 17087 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:08:09.970216 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:35761
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:10.095367 17098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:10.095676 17098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:10.095726 17098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:10.100142 17098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:10.100226 17098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:10.100293 17098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:08:10.105809 17098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:35761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:08:10.106966 17098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:10.108064 17098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:10.118440 17103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:10.118430 17104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:10.118723 17098 server_base.cc:1034] running on GCE node
W20260620 19:08:10.118430 17106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:10.119270 17098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:10.120023 17098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:10.123446 17098 hybrid_clock.cc:648] HybridClock initialized: now 1781982490123393 us; error 90 us; skew 500 ppm
Jun 20 19:08:10 dist-test-slave-6wrf krb5kdc[16991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:10.127446 17098 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:08:10.128492 17098 webserver.cc:460] Webserver started at http://127.1.154.129:42399/ using document root <none> and password file <none>
I20260620 19:08:10.129146 17098 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:10.129252 17098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:10.129503 17098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:10.131630 17098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "f4e9dea926bd4ec69d12606ca66f348a"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-6wrf"
server_key: "d9ddadce4a69e31f1a3d294236392abc"
server_key_iv: "a44f78c5d7ad63617059b1a57f5fc2c5"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.132228 17098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "f4e9dea926bd4ec69d12606ca66f348a"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-6wrf"
server_key: "d9ddadce4a69e31f1a3d294236392abc"
server_key_iv: "a44f78c5d7ad63617059b1a57f5fc2c5"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.136134 17098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 19:08:10.139319 17113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.141024 17098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:08:10.141223 17098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "f4e9dea926bd4ec69d12606ca66f348a"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-6wrf"
server_key: "d9ddadce4a69e31f1a3d294236392abc"
server_key_iv: "a44f78c5d7ad63617059b1a57f5fc2c5"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.141360 17098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:10.156008 17098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:10.160488 17098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:10.160754 17098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:10.161489 17098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:10.162472 17098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:10.162549 17098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.162617 17098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:10.162671 17098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.173414 17098 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:38577
I20260620 19:08:10.173482 17226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:38577 every 8 connection(s)
I20260620 19:08:10.174775 17098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:08:10.178898 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 17098
I20260620 19:08:10.179068 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:08:10.179379 1642 external_mini_cluster.cc:1428] Setting key f3f787e46043c935301703681c130096
Jun 20 19:08:10 dist-test-slave-6wrf krb5kdc[16991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
I20260620 19:08:10.192502 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.177164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:33333 (local address 127.1.154.190:35761)
0620 19:08:10.177321 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.177325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.178318 (+ 993us) server_negotiation.cc:407] Connection header received
0620 19:08:10.180020 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.180026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.180099 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.180173 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.181754 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.182310 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.183622 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.184058 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.184231 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.187550 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.187553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.187555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:10.187584 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.189495 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:10.190461 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:10.190465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:10.190468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:10.190664 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:10.191204 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:10.191210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:10.191212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:10.191428 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.191562 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.192141 (+ 579us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.192303 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
I20260620 19:08:10.193432 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.177399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:33333)
0620 19:08:10.178141 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.178188 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.179763 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.180386 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.180393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.180858 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:10.181543 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.181564 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.182452 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.182458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.183466 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.183475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.184431 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.184438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.184700 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.185777 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:10.185805 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:10.187352 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:10.189784 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:10.189794 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:10.189825 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:10.190305 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:10.190838 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:10.190843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:10.190846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:10.191068 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:10.191561 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:10.191571 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:10.191860 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.192081 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.192460 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":160,"threads_started":1}
I20260620 19:08:10.195050 17227 heartbeater.cc:347] Connected to a master server at 127.1.154.190:35761
I20260620 19:08:10.195343 17227 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:10.195856 17227 heartbeater.cc:511] Master 127.1.154.190:35761 requested a full tablet report, sending...
I20260620 19:08:10.197719 17036 ts_manager.cc:194] Registered new tserver with Master: f4e9dea926bd4ec69d12606ca66f348a (127.1.154.129:38577)
I20260620 19:08:10.199491 17036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:33333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:08:10.213364 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:35761
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:10.338797 17232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:10.339074 17232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:10.339118 17232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:10.344688 17232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:10.344767 17232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:10.344830 17232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:08:10.351958 17232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:35761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:08:10.353993 17232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:10.355461 17232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:10.367064 17237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:10.367111 17238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:10.367064 17240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:10.367331 17232 server_base.cc:1034] running on GCE node
I20260620 19:08:10.368046 17232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:10.368757 17232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:10.369943 17232 hybrid_clock.cc:648] HybridClock initialized: now 1781982490369927 us; error 32 us; skew 500 ppm
Jun 20 19:08:10 dist-test-slave-6wrf krb5kdc[16991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:10.373996 17232 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:08:10.375415 17232 webserver.cc:460] Webserver started at http://127.1.154.130:39725/ using document root <none> and password file <none>
I20260620 19:08:10.375972 17232 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:10.376035 17232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:10.376214 17232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:10.378362 17232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-6wrf"
server_key: "23c4325a380fdb3835fe43b33e57e841"
server_key_iv: "4d3212760d2270f23f879a7c00eed645"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.378880 17232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-6wrf"
server_key: "23c4325a380fdb3835fe43b33e57e841"
server_key_iv: "4d3212760d2270f23f879a7c00eed645"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.383754 17232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 19:08:10.386950 17247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.388562 17232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:08:10.388690 17232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-6wrf"
server_key: "23c4325a380fdb3835fe43b33e57e841"
server_key_iv: "4d3212760d2270f23f879a7c00eed645"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.388794 17232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:10.414621 17232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:10.418241 17232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:10.418462 17232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:10.419504 17232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:10.420413 17232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:10.420461 17232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.420542 17232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:10.420583 17232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.430797 17232 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:33703
I20260620 19:08:10.430809 17360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:33703 every 8 connection(s)
I20260620 19:08:10.432404 17232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:08:10.432941 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 17232
I20260620 19:08:10.433070 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:08:10.433415 1642 external_mini_cluster.cc:1428] Setting key 09ee18701225f1121fd46999147dc26b
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
Jun 20 19:08:10 dist-test-slave-6wrf krb5kdc[16991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
I20260620 19:08:10.451988 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.434781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:49591 (local address 127.1.154.190:35761)
0620 19:08:10.435074 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.435079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.436211 (+ 1132us) server_negotiation.cc:407] Connection header received
0620 19:08:10.437571 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.437576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.437673 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.437783 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.439323 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.439981 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.441959 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.442179 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.442277 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.446639 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.446643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.446646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:10.446685 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.448503 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:10.449557 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:10.449561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:10.449563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:10.449742 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:10.450262 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:10.450265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:10.450266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:10.450538 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.450661 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.451652 (+ 991us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.451812 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 19:08:10.452847 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.435128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:49591)
0620 19:08:10.435977 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.436034 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.437341 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.437937 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.437944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.438417 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:10.439150 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.439172 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.440140 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.440147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.441773 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.441788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.442364 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.442370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.442585 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.443762 (+ 1177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:10.443789 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:10.446311 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:10.448757 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:10.448765 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:10.448815 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:10.449419 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:10.449851 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:10.449858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:10.449863 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:10.450085 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:10.450804 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:10.450813 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:10.451350 (+ 537us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.451550 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.451874 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":126,"threads_started":1}
I20260620 19:08:10.454448 17361 heartbeater.cc:347] Connected to a master server at 127.1.154.190:35761
I20260620 19:08:10.454800 17361 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:10.455361 17361 heartbeater.cc:511] Master 127.1.154.190:35761 requested a full tablet report, sending...
I20260620 19:08:10.456568 17036 ts_manager.cc:194] Registered new tserver with Master: 0c8f9d494ae24bf3b2d1e1ba50d1755f (127.1.154.130:33703)
I20260620 19:08:10.457397 17036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:49591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:08:10.470547 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:35761
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:10.589316 17366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:10.589663 17366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:10.589715 17366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:10.595428 17366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:10.595515 17366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:10.595582 17366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:08:10.602596 17366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:35761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:08:10.604270 17366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:10.605717 17366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:10.618739 17374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:10.618703 17371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:10.618898 17366 server_base.cc:1034] running on GCE node
W20260620 19:08:10.618721 17372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:10.619547 17366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:10.620292 17366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:10.621757 17366 hybrid_clock.cc:648] HybridClock initialized: now 1781982490621680 us; error 102 us; skew 500 ppm
Jun 20 19:08:10 dist-test-slave-6wrf krb5kdc[16991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:10.625819 17366 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:08:10.627177 17366 webserver.cc:460] Webserver started at http://127.1.154.131:44341/ using document root <none> and password file <none>
I20260620 19:08:10.627856 17366 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:10.627930 17366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:10.628167 17366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:10.630535 17366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "f8e6745d51754d8996377f5394072585"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-6wrf"
server_key: "561f4f9a797a3be5f6c0944e53c60e78"
server_key_iv: "926a5acba1b6e06f52065993e712719d"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.631121 17366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "f8e6745d51754d8996377f5394072585"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-6wrf"
server_key: "561f4f9a797a3be5f6c0944e53c60e78"
server_key_iv: "926a5acba1b6e06f52065993e712719d"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.635821 17366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:08:10.639881 17381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.641678 17366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20260620 19:08:10.641819 17366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "f8e6745d51754d8996377f5394072585"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-6wrf"
server_key: "561f4f9a797a3be5f6c0944e53c60e78"
server_key_iv: "926a5acba1b6e06f52065993e712719d"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.641990 17366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:10.656356 17366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:10.660849 17366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:10.661330 17366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:10.662140 17366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:10.663370 17366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:10.663426 17366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.663471 17366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:10.663537 17366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.673839 17366 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:42011
I20260620 19:08:10.673890 17494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:42011 every 8 connection(s)
I20260620 19:08:10.674866 17366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:08:10.678884 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 17366
I20260620 19:08:10.679037 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:08:10.679366 1642 external_mini_cluster.cc:1428] Setting key 7c3565b0535011cfdceabe6479ec2452
Jun 20 19:08:10 dist-test-slave-6wrf krb5kdc[16991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:08:10.692159 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.676973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:42583 (local address 127.1.154.190:35761)
0620 19:08:10.677165 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.677170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.677998 (+ 828us) server_negotiation.cc:407] Connection header received
0620 19:08:10.679818 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.679821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.679882 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.680096 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.681676 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.682530 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.684702 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.684971 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.685077 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.688307 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.688311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.688313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:10.688343 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.690001 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:10.690552 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:10.690557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:10.690559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:10.690635 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:10.691018 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:10.691022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:10.691024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:10.691223 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.691332 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.691866 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.691996 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 19:08:10.693214 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.677233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:42583)
0620 19:08:10.677836 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.677880 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.679510 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.680235 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.680246 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.680729 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:10.681407 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.681429 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.682758 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.682762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.684536 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.684555 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.685189 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.685193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.685428 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.686386 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:10.686402 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:10.688081 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:10.690133 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:10.690139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:10.690163 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:10.690435 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:10.690738 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:10.690743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:10.690745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:10.690916 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:10.691365 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:10.691377 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:10.691619 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.691821 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.692160 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":119,"threads_started":1}
I20260620 19:08:10.694605 17495 heartbeater.cc:347] Connected to a master server at 127.1.154.190:35761
I20260620 19:08:10.694897 17495 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:10.695415 17495 heartbeater.cc:511] Master 127.1.154.190:35761 requested a full tablet report, sending...
I20260620 19:08:10.696903 17036 ts_manager.cc:194] Registered new tserver with Master: f8e6745d51754d8996377f5394072585 (127.1.154.131:42011)
I20260620 19:08:10.697734 17036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:42583
I20260620 19:08:10.707942 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:08:10.719619 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.710771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60250 (local address 127.1.154.190:35761)
0620 19:08:10.710939 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.710946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.710965 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:08:10.711021 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.711024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.711064 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.711159 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.711878 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.712417 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.713260 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.713488 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.713602 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.715804 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.715808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.715810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:10.715836 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.717730 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:10.718225 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:10.718229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:10.718231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:10.718314 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:10.718674 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:10.718679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:10.718681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:10.718927 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.719075 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.719314 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.719442 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:08:10.723590 17036 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:08:10.726763 17036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:08:10.743844 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.737012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:42011 (local address 127.0.0.1:36532)
0620 19:08:10.737483 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.737536 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.737720 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.738917 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.738920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.738945 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:10.739079 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.739085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.740484 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.740488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.741766 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.741778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.743351 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.743356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.743460 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.743483 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.743578 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.743653 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":52,"thread_start_us":90,"threads_started":1}
I20260620 19:08:10.744006 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.736716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:38577 (local address 127.0.0.1:50384)
0620 19:08:10.737483 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.737536 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.737720 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.738288 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.738294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.738314 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:10.738485 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.738492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.739981 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.739984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.742093 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.742121 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.743586 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.743592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.743693 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.743725 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.743799 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.743874 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":400,"thread_start_us":89,"threads_started":1}
I20260620 19:08:10.744251 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.737440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:33703 (local address 127.0.0.1:35064)
0620 19:08:10.738193 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.738208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.738303 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.738765 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.738768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.738789 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:10.738885 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.738892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.739912 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.739917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.742209 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.742219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.743077 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.743081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.743188 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.743228 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.743349 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.743471 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":630,"thread_start_us":92,"threads_started":1}
I20260620 19:08:10.744421 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.737821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.1.154.130:33703)
0620 19:08:10.738234 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.738250 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.738272 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:08:10.738444 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.738449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.738617 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.738755 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:10.739011 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.739766 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.742348 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.742932 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.743078 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.743252 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.743708 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.743807 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":116,"threads_started":1}
I20260620 19:08:10.744428 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.736891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.1.154.129:38577)
0620 19:08:10.737274 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.737283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.737654 (+ 371us) server_negotiation.cc:407] Connection header received
0620 19:08:10.737879 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.737887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.738089 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.738228 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:10.738625 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.739865 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.742344 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.743358 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.743462 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.743579 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.744180 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.744292 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":91,"threads_started":1}
I20260620 19:08:10.744794 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.737276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.1.154.131:42011)
0620 19:08:10.738016 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.738026 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.738045 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:08:10.738444 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.738449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.738617 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.738735 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:10.739217 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.740312 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.742135 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.743199 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.743354 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.743501 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.744505 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.744624 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":131,"threads_started":1}
I20260620 19:08:10.747414 17161 tablet_service.cc:1467] Processing CreateTablet for tablet b25b9cc86a0b4c9f9e676bc4dae6bd52 (DEFAULT_TABLE table=test-table [id=2f77d782245d49e881beb6566d9e1042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:10.747423 17429 tablet_service.cc:1467] Processing CreateTablet for tablet b25b9cc86a0b4c9f9e676bc4dae6bd52 (DEFAULT_TABLE table=test-table [id=2f77d782245d49e881beb6566d9e1042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:10.747435 17294 tablet_service.cc:1467] Processing CreateTablet for tablet b25b9cc86a0b4c9f9e676bc4dae6bd52 (DEFAULT_TABLE table=test-table [id=2f77d782245d49e881beb6566d9e1042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:10.748472 17161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b25b9cc86a0b4c9f9e676bc4dae6bd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:10.748495 17429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b25b9cc86a0b4c9f9e676bc4dae6bd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:10.748490 17294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b25b9cc86a0b4c9f9e676bc4dae6bd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:10.756505 17512 tablet_bootstrap.cc:492] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585: Bootstrap starting.
I20260620 19:08:10.757130 17513 tablet_bootstrap.cc:492] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Bootstrap starting.
I20260620 19:08:10.757853 17514 tablet_bootstrap.cc:492] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: Bootstrap starting.
I20260620 19:08:10.760365 17513 tablet_bootstrap.cc:654] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:10.760687 17512 tablet_bootstrap.cc:654] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:10.760687 17514 tablet_bootstrap.cc:654] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:10.761322 17513 log.cc:824] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:10.762173 17514 log.cc:824] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:10.762461 17512 log.cc:824] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:10.764591 17513 tablet_bootstrap.cc:492] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f: No bootstrap required, opened a new log
I20260620 19:08:10.765055 17513 ts_tablet_manager.cc:1397] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 19:08:10.765694 17514 tablet_bootstrap.cc:492] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: No bootstrap required, opened a new log
I20260620 19:08:10.765731 17512 tablet_bootstrap.cc:492] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585: No bootstrap required, opened a new log
I20260620 19:08:10.765940 17512 ts_tablet_manager.cc:1397] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 19:08:10.765941 17514 ts_tablet_manager.cc:1397] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 19:08:10.768630 17513 raft_consensus.cc:348] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } } peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } }
I20260620 19:08:10.768959 17513 raft_consensus.cc:374] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:10.769021 17513 raft_consensus.cc:729] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8f9d494ae24bf3b2d1e1ba50d1755f, State: Initialized, Role: FOLLOWER
I20260620 19:08:10.769517 17513 consensus_queue.cc:260] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } } peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } }
I20260620 19:08:10.770310 17361 heartbeater.cc:503] Master 127.1.154.190:35761 was elected leader, sending a full tablet report...
I20260620 19:08:10.770478 17512 raft_consensus.cc:348] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } } peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } }
I20260620 19:08:10.770825 17513 ts_tablet_manager.cc:1428] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:08:10.770957 17512 raft_consensus.cc:374] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:10.771216 17512 raft_consensus.cc:729] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e6745d51754d8996377f5394072585, State: Initialized, Role: FOLLOWER
I20260620 19:08:10.771499 17514 raft_consensus.cc:348] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } } peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } }
I20260620 19:08:10.771934 17514 raft_consensus.cc:374] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:10.771981 17514 raft_consensus.cc:729] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e9dea926bd4ec69d12606ca66f348a, State: Initialized, Role: FOLLOWER
I20260620 19:08:10.771879 17512 consensus_queue.cc:260] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } } peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } }
I20260620 19:08:10.772543 17514 consensus_queue.cc:260] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } } peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } }
I20260620 19:08:10.773326 17512 ts_tablet_manager.cc:1428] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 19:08:10.773734 17514 ts_tablet_manager.cc:1428] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260620 19:08:10.773823 17495 heartbeater.cc:503] Master 127.1.154.190:35761 was elected leader, sending a full tablet report...
I20260620 19:08:10.774098 17227 heartbeater.cc:503] Master 127.1.154.190:35761 was elected leader, sending a full tablet report...
W20260620 19:08:10.926023 17496 tablet.cc:2369] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:08:10.926461 17228 tablet.cc:2369] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:08:10.933990 17362 tablet.cc:2369] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:08:10.978747 17518 raft_consensus.cc:482] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:10.979120 17518 raft_consensus.cc:504] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } } peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } }
I20260620 19:08:10.980840 17518 leader_election.cc:288] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4e9dea926bd4ec69d12606ca66f348a (127.1.154.129:38577), f8e6745d51754d8996377f5394072585 (127.1.154.131:42011)
I20260620 19:08:10.985765 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.981455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:57411 (local address 127.1.154.131:42011)
0620 19:08:10.981616 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.981622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.981995 (+ 373us) server_negotiation.cc:407] Connection header received
0620 19:08:10.982177 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.982180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.982242 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.982346 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:10.982746 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.983400 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.984648 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.985057 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.985154 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.985222 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.985503 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.985584 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:08:10.985810 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.981180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:38577 (local address 127.1.154.130:57555)
0620 19:08:10.981812 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.981829 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.982019 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.982306 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.982309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.982325 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:10.982508 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.982515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.983471 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.983477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.984505 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.984515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.985275 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.985279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.985363 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.985380 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.985486 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.985559 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":37,"thread_start_us":141,"threads_started":1}
I20260620 19:08:10.985810 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.981303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:57555 (local address 127.1.154.129:38577)
0620 19:08:10.981591 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.981597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.981949 (+ 352us) server_negotiation.cc:407] Connection header received
0620 19:08:10.982136 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.982140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.982191 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.982286 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:10.982617 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.983353 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.984655 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.985138 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.985267 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.985327 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.985538 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.985672 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 19:08:10.985837 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.981288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:42011 (local address 127.1.154.130:57411)
0620 19:08:10.981858 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.981872 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.982044 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.982434 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.982438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.982451 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:10.982614 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.982622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.983534 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.983537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.984501 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.984511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.985199 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.985205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.985318 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.985352 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.985483 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.985570 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":24,"spinlock_wait_cycles":768,"thread_start_us":220,"threads_started":1}
I20260620 19:08:10.986690 17181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25b9cc86a0b4c9f9e676bc4dae6bd52" candidate_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4e9dea926bd4ec69d12606ca66f348a" is_pre_election: true
I20260620 19:08:10.986837 17449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25b9cc86a0b4c9f9e676bc4dae6bd52" candidate_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e6745d51754d8996377f5394072585" is_pre_election: true
I20260620 19:08:10.987073 17181 raft_consensus.cc:2454] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8f9d494ae24bf3b2d1e1ba50d1755f in term 0.
I20260620 19:08:10.987190 17449 raft_consensus.cc:2454] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8f9d494ae24bf3b2d1e1ba50d1755f in term 0.
I20260620 19:08:10.987651 17249 leader_election.cc:302] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c8f9d494ae24bf3b2d1e1ba50d1755f, f4e9dea926bd4ec69d12606ca66f348a, f8e6745d51754d8996377f5394072585; no voters:
I20260620 19:08:10.987965 17518 raft_consensus.cc:2781] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:08:10.988058 17518 raft_consensus.cc:482] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:08:10.988112 17518 raft_consensus.cc:3037] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:10.989467 17518 raft_consensus.cc:504] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } } peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } }
I20260620 19:08:10.990453 17518 leader_election.cc:288] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [CANDIDATE]: Term 1 election: Requested vote from peers f4e9dea926bd4ec69d12606ca66f348a (127.1.154.129:38577), f8e6745d51754d8996377f5394072585 (127.1.154.131:42011)
I20260620 19:08:10.990996 17449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25b9cc86a0b4c9f9e676bc4dae6bd52" candidate_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e6745d51754d8996377f5394072585"
I20260620 19:08:10.991022 17181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25b9cc86a0b4c9f9e676bc4dae6bd52" candidate_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4e9dea926bd4ec69d12606ca66f348a"
I20260620 19:08:10.991169 17449 raft_consensus.cc:3037] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:10.991189 17181 raft_consensus.cc:3037] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:10.992249 17449 raft_consensus.cc:2454] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8f9d494ae24bf3b2d1e1ba50d1755f in term 1.
I20260620 19:08:10.992328 17181 raft_consensus.cc:2454] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8f9d494ae24bf3b2d1e1ba50d1755f in term 1.
I20260620 19:08:10.992645 17251 leader_election.cc:302] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8f9d494ae24bf3b2d1e1ba50d1755f, f8e6745d51754d8996377f5394072585; no voters:
I20260620 19:08:10.992892 17518 raft_consensus.cc:2781] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:10.993211 17518 raft_consensus.cc:686] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 1 LEADER]: Becoming Leader. State: Replica: 0c8f9d494ae24bf3b2d1e1ba50d1755f, State: Running, Role: LEADER
I20260620 19:08:10.993569 17518 consensus_queue.cc:237] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } } peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } }
I20260620 19:08:10.996956 17036 catalog_manager.cc:5495] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8f9d494ae24bf3b2d1e1ba50d1755f (127.1.154.130). New cstate: current_term: 1 leader_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } health_report { overall_health: HEALTHY } } }
I20260620 19:08:11.054309 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:11.049448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35068 (local address 127.1.154.130:33703)
0620 19:08:11.049639 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:11.049644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:11.049660 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:08:11.049774 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:11.049777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:11.049852 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:11.049985 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:08:11.050444 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.051009 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.052296 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.052566 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.052674 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:11.053125 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:08:11.053632 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:08:11.053744 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:11.054051 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 19:08:11.054137 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 19:08:11.062232 17181 raft_consensus.cc:1261] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Refusing update from remote peer 0c8f9d494ae24bf3b2d1e1ba50d1755f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:11.062232 17449 raft_consensus.cc:1261] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585 [term 1 FOLLOWER]: Refusing update from remote peer 0c8f9d494ae24bf3b2d1e1ba50d1755f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:11.063438 17523 consensus_queue.cc:1035] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:11.063757 17518 consensus_queue.cc:1035] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:11.072705 17526 mvcc.cc:205] Tried to move back new op lower bound from 7299000283381919744 to 7299000283112980480. Current Snapshot: MvccSnapshot[applied={T|T < 7299000283381919744}]
I20260620 19:08:11.074086 17528 mvcc.cc:205] Tried to move back new op lower bound from 7299000283381919744 to 7299000283112980480. Current Snapshot: MvccSnapshot[applied={T|T < 7299000283381919744}]
I20260620 19:08:11.086732 17035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:11.086942 17035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:11.090041 17035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Sending DeleteTablet for 3 replicas of tablet b25b9cc86a0b4c9f9e676bc4dae6bd52
I20260620 19:08:11.091351 17429 tablet_service.cc:1514] Processing DeleteTablet for tablet b25b9cc86a0b4c9f9e676bc4dae6bd52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:11 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:36532
I20260620 19:08:11.091432 17161 tablet_service.cc:1514] Processing DeleteTablet for tablet b25b9cc86a0b4c9f9e676bc4dae6bd52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:11 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:50384
I20260620 19:08:11.091545 17294 tablet_service.cc:1514] Processing DeleteTablet for tablet b25b9cc86a0b4c9f9e676bc4dae6bd52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:11 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:35064
I20260620 19:08:11.091895 17536 tablet_replica.cc:330] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585: stopping tablet replica
I20260620 19:08:11.091894 17537 tablet_replica.cc:330] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: stopping tablet replica
I20260620 19:08:11.092221 17537 raft_consensus.cc:2229] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:11.092221 17536 raft_consensus.cc:2229] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:11.092406 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 17098
I20260620 19:08:11.093299 17536 raft_consensus.cc:2258] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:11.094509 17538 tablet_replica.cc:330] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f: stopping tablet replica
I20260620 19:08:11.094866 17538 raft_consensus.cc:2229] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:08:11.095254 17536 ts_tablet_manager.cc:1905] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:11.095394 17538 raft_consensus.cc:2258] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:11.096853 17538 ts_tablet_manager.cc:1905] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:11.098441 17536 ts_tablet_manager.cc:1918] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:11.098552 17536 log.cc:1196] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/b25b9cc86a0b4c9f9e676bc4dae6bd52
I20260620 19:08:11.099180 17536 ts_tablet_manager.cc:1939] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f8e6745d51754d8996377f5394072585: Deleting consensus metadata
I20260620 19:08:11.101153 17538 ts_tablet_manager.cc:1918] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260620 19:08:11.101343 17021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:38577 (error 108)
I20260620 19:08:11.101378 17023 catalog_manager.cc:4841] TS f8e6745d51754d8996377f5394072585 (127.1.154.131:42011): tablet b25b9cc86a0b4c9f9e676bc4dae6bd52 (table test-table [id=2f77d782245d49e881beb6566d9e1042]) successfully deleted
I20260620 19:08:11.101647 17538 log.cc:1196] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/b25b9cc86a0b4c9f9e676bc4dae6bd52
I20260620 19:08:11.102334 17538 ts_tablet_manager.cc:1939] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Deleting consensus metadata
I20260620 19:08:11.102744 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:11.102388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:38577 (local address 127.0.0.1:50390)
0620 19:08:11.102529 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:11.102604 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:38577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 19:08:11.102869 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 17232
W20260620 19:08:11.102942 17021 catalog_manager.cc:4584] TS f4e9dea926bd4ec69d12606ca66f348a (127.1.154.129:38577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b25b9cc86a0b4c9f9e676bc4dae6bd52: Network error: Client connection negotiation failed: client connection to 127.1.154.129:38577: connect: Connection refused (error 111)
I20260620 19:08:11.109819 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 17366
I20260620 19:08:11.111670 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:11.110918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:33703 (local address 127.0.0.1:35070)
0620 19:08:11.111470 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:11.111550 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.130:33703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427}
I20260620 19:08:11.116328 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 17004
I20260620 19:08:31.124948 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:35761
--webserver_interface=127.1.154.190
--webserver_port=35601
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:35761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:31.258265 17540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:31.258735 17540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:31.258795 17540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:31.263636 17540 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:08:31.263739 17540 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:08:31.263757 17540 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:08:31.263772 17540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:31.263793 17540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:31.263813 17540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:31.263836 17540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:31.263854 17540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:31.270319 17540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:35761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:35761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=35601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:08:31.271836 17540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:31.273098 17540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:31.290063 17549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:31.290165 17540 server_base.cc:1034] running on GCE node
W20260620 19:08:31.290202 17546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:31.290167 17547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:31.291193 17540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:31.292601 17540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:31.293812 17540 hybrid_clock.cc:648] HybridClock initialized: now 1781982511293775 us; error 62 us; skew 500 ppm
Jun 20 19:08:31 dist-test-slave-6wrf krb5kdc[16991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982511, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:31.297089 17540 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:08:31.298163 17540 webserver.cc:460] Webserver started at http://127.1.154.190:35601/ using document root <none> and password file <none>
I20260620 19:08:31.298815 17540 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:31.298945 17540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:31.303548 17540 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:08:31.306669 17556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:31.308378 17540 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:08:31.308566 17540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "1fbc3766918b45ceb67294eb0ae91f3d"
format_stamp: "Formatted at 2026-06-20 19:08:09 on dist-test-slave-6wrf"
server_key: "6eee58b40b2e04583d146084f7691338"
server_key_iv: "f86725cc54471aa6e64050bc64293ba8"
server_key_version: "encryptionkey@0"
I20260620 19:08:31.309314 17540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:31.325119 17540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:31.329046 17540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:31.329289 17540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:31.337885 17540 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:35761
I20260620 19:08:31.338249 17608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:35761 every 8 connection(s)
I20260620 19:08:31.339136 17540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:31.344260 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 17540
I20260620 19:08:31.344854 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:38577
--local_ip_for_outbound_sockets=127.1.154.129
--tserver_master_addrs=127.1.154.190:35761
--webserver_port=42399
--webserver_interface=127.1.154.129
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:08:31.345655 17609 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:08:31.348459 17609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Bootstrap starting.
I20260620 19:08:31.369858 17609 log.cc:824] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:31.380450 17609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:08:31.380995 17609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Bootstrap complete.
I20260620 19:08:31.385931 17609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } }
I20260620 19:08:31.386447 17609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fbc3766918b45ceb67294eb0ae91f3d, State: Initialized, Role: FOLLOWER
I20260620 19:08:31.387037 17609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } }
I20260620 19:08:31.387189 17609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:31.387274 17609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:31.387392 17609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:08:31.389998 17609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } }
I20260620 19:08:31.390444 17609 leader_election.cc:302] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fbc3766918b45ceb67294eb0ae91f3d; no voters:
I20260620 19:08:31.390821 17609 leader_election.cc:288] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:08:31.391218 17613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:08:31.391741 17613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 2 LEADER]: Becoming Leader. State: Replica: 1fbc3766918b45ceb67294eb0ae91f3d, State: Running, Role: LEADER
I20260620 19:08:31.392313 17609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:31.392423 17613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } }
I20260620 19:08:31.394294 17614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } } }
I20260620 19:08:31.394470 17614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:31.394325 17615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fbc3766918b45ceb67294eb0ae91f3d. Latest consensus state: current_term: 2 leader_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } } }
I20260620 19:08:31.394634 17615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:31.395043 17623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:31.400048 17623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:31.400740 17623 catalog_manager.cc:1259] Loaded cluster ID: fb6d796b51a7402e9ef44d36bfc79661
I20260620 19:08:31.400815 17623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:31.403122 17623 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:31.403766 17623 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Loaded TSK: 0
I20260620 19:08:31.415668 17623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Generated new TSK 1
I20260620 19:08:31.416026 17623 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:08:31.487401 17611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:31.487752 17611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:31.487835 17611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:31.492491 17611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:31.492600 17611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:31.492709 17611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:08:31.499363 17611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:38577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=42399
--tserver_master_addrs=127.1.154.190:35761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:08:31.500771 17611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:31.502079 17611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:31.514787 17637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:31.514787 17639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:31.514882 17636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:31.515486 17611 server_base.cc:1034] running on GCE node
I20260620 19:08:31.516040 17611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:31.517020 17611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:31.518240 17611 hybrid_clock.cc:648] HybridClock initialized: now 1781982511518211 us; error 51 us; skew 500 ppm
Jun 20 19:08:31 dist-test-slave-6wrf krb5kdc[16991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982511, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:31.521570 17611 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:08:31.522929 17611 webserver.cc:460] Webserver started at http://127.1.154.129:42399/ using document root <none> and password file <none>
I20260620 19:08:31.523640 17611 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:31.523728 17611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:31.528563 17611 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:08:31.531244 17646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:31.532846 17611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:08:31.533030 17611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "f4e9dea926bd4ec69d12606ca66f348a"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-6wrf"
server_key: "d9ddadce4a69e31f1a3d294236392abc"
server_key_iv: "a44f78c5d7ad63617059b1a57f5fc2c5"
server_key_version: "encryptionkey@0"
I20260620 19:08:31.533787 17611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:31.555172 17611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:31.559326 17611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:31.559595 17611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:31.560369 17611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:31.561877 17653 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:08:31.566359 17611 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:08:31.566519 17611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260620 19:08:31.566651 17611 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:08:31.568979 17611 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:08:31.569064 17611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 19:08:31.569437 17653 tablet_bootstrap.cc:492] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: Bootstrap starting.
I20260620 19:08:31.581916 17611 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:38577
I20260620 19:08:31.582160 17760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:38577 every 8 connection(s)
I20260620 19:08:31.583295 17611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:08:31.584220 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 17611
I20260620 19:08:31.584728 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:33703
--local_ip_for_outbound_sockets=127.1.154.130
--tserver_master_addrs=127.1.154.190:35761
--webserver_port=39725
--webserver_interface=127.1.154.130
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:08:31.589416 17653 log.cc:824] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:08:31 dist-test-slave-6wrf krb5kdc[16991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982511, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:08:31.600585 17653 tablet_bootstrap.cc:492] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:08:31.601048 17653 tablet_bootstrap.cc:492] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: Bootstrap complete.
I20260620 19:08:31.602650 17653 ts_tablet_manager.cc:1397] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: Time spent bootstrapping tablet: real 0.033s user 0.027s sys 0.004s
I20260620 19:08:31.604321 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:31.586709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:41465)
0620 19:08:31.587607 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:31.587672 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:31.588713 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:31.590301 (+ 1588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:31.590309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:31.590733 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:31.591537 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:31.591558 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.593206 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.593209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:31.594496 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:31.594508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.595427 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.595431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:31.595622 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:31.596352 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:31.596371 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:31.598143 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:31.600484 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:31.600490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:31.600506 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:31.600894 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:31.601341 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:31.601346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:31.601348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:31.601509 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:31.602209 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:31.602217 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:31.602514 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 19:08:31.602789 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 19:08:31.603205 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":184,"threads_started":1}
I20260620 19:08:31.604321 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:31.586717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:41465 (local address 127.1.154.190:35761)
0620 19:08:31.587262 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:31.587285 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:31.587819 (+ 534us) server_negotiation.cc:407] Connection header received
0620 19:08:31.589279 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:31.589297 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:31.589779 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:31.590188 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:31.591755 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.593012 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.594645 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.595268 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.595459 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:31.598401 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:31.598408 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:31.598422 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:31.598459 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:31.600339 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:31.601035 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:31.601041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:31.601045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:31.601239 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:31.601647 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:31.601651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:31.601653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:31.602086 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:31.602247 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:31.602849 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 19:08:31.603199 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":156,"threads_started":1}
I20260620 19:08:31.605706 17653 raft_consensus.cc:348] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } } peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } }
I20260620 19:08:31.606261 17653 raft_consensus.cc:729] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e9dea926bd4ec69d12606ca66f348a, State: Initialized, Role: FOLLOWER
I20260620 19:08:31.606804 17653 consensus_queue.cc:260] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } } peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } }
I20260620 19:08:31.607772 17761 heartbeater.cc:347] Connected to a master server at 127.1.154.190:35761
I20260620 19:08:31.607923 17653 ts_tablet_manager.cc:1428] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:08:31.608111 17761 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:31.608817 17761 heartbeater.cc:511] Master 127.1.154.190:35761 requested a full tablet report, sending...
I20260620 19:08:31.612504 17569 ts_manager.cc:194] Registered new tserver with Master: f4e9dea926bd4ec69d12606ca66f348a (127.1.154.129:38577)
I20260620 19:08:31.617278 17569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:41465
I20260620 19:08:31.620849 17761 heartbeater.cc:503] Master 127.1.154.190:35761 was elected leader, sending a full tablet report...
Jun 20 19:08:31 dist-test-slave-6wrf krb5kdc[16991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982511, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for kudu/127.1.154.129@KRBTEST.COM
I20260620 19:08:31.630597 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:31.616552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:38577 (local address 127.0.0.1:39356)
0620 19:08:31.617451 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:31.617517 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:31.617764 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:31.618635 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:31.618642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:31.619119 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:31.619300 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:31.619308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.620847 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.620851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:31.621893 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:31.621904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.622827 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.622831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:31.622941 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:31.623640 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:31.623655 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:31.625836 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:31.628358 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:31.628367 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:31.628372 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:31.628687 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:31.629095 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:31.629099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:31.629101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:31.629209 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:31.629866 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:31.629873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:31.630023 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 19:08:31.630174 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 19:08:31.630395 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":211,"threads_started":1}
I20260620 19:08:31.630656 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:31.616638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39356 (local address 127.1.154.129:38577)
0620 19:08:31.617088 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:31.617100 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:31.617719 (+ 619us) server_negotiation.cc:407] Connection header received
0620 19:08:31.618003 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:31.618010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:31.618280 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:31.618472 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:31.619468 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.620678 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.622036 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.622670 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.622807 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:31.626132 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:31.626143 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:31.626160 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:31.626192 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:31.628223 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:31.628874 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:31.628879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:31.628881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:31.628978 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:31.629332 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:31.629335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:31.629336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:31.629739 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:31.629869 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:31.630297 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 19:08:31.630489 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":348,"thread_start_us":128,"threads_started":1}
I20260620 19:08:31.632388 17695 tablet_service.cc:1514] Processing DeleteTablet for tablet b25b9cc86a0b4c9f9e676bc4dae6bd52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:11 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:39356
I20260620 19:08:31.633481 17770 tablet_replica.cc:330] stopping tablet replica
I20260620 19:08:31.633884 17770 raft_consensus.cc:2229] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:31.634203 17770 raft_consensus.cc:2258] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:31.635378 17770 ts_tablet_manager.cc:1905] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:31.640097 17770 ts_tablet_manager.cc:1918] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:31.640221 17770 log.cc:1196] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal/wals/b25b9cc86a0b4c9f9e676bc4dae6bd52
I20260620 19:08:31.640517 17770 ts_tablet_manager.cc:1939] T b25b9cc86a0b4c9f9e676bc4dae6bd52 P f4e9dea926bd4ec69d12606ca66f348a: Deleting consensus metadata
I20260620 19:08:31.643100 17558 catalog_manager.cc:4841] TS f4e9dea926bd4ec69d12606ca66f348a (127.1.154.129:38577): tablet b25b9cc86a0b4c9f9e676bc4dae6bd52 (table test-table [id=2f77d782245d49e881beb6566d9e1042]) successfully deleted
W20260620 19:08:31.718076 17763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:31.718377 17763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:31.718452 17763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:31.722500 17763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:31.722584 17763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:31.722676 17763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:08:31.728159 17763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:33703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=39725
--tserver_master_addrs=127.1.154.190:35761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:08:31.729350 17763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:31.730468 17763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:31.741801 17775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:31.741816 17776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:31.741957 17763 server_base.cc:1034] running on GCE node
W20260620 19:08:31.741845 17778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:31.742713 17763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:31.743690 17763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:31.744940 17763 hybrid_clock.cc:648] HybridClock initialized: now 1781982511744908 us; error 49 us; skew 500 ppm
Jun 20 19:08:31 dist-test-slave-6wrf krb5kdc[16991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982511, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:31.748558 17763 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:08:31.749789 17763 webserver.cc:460] Webserver started at http://127.1.154.130:39725/ using document root <none> and password file <none>
I20260620 19:08:31.750492 17763 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:31.750583 17763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:31.756009 17763 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 19:08:31.759243 17785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:31.761322 17763 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:08:31.761497 17763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-6wrf"
server_key: "23c4325a380fdb3835fe43b33e57e841"
server_key_iv: "4d3212760d2270f23f879a7c00eed645"
server_key_version: "encryptionkey@0"
I20260620 19:08:31.762351 17763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:31.784952 17763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:31.789346 17763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:31.789587 17763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:31.790520 17763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:31.791604 17763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:31.791666 17763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:31.791715 17763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:31.791731 17763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:31.803836 17763 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:33703
I20260620 19:08:31.803937 17898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:33703 every 8 connection(s)
I20260620 19:08:31.805235 17763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:08:31.815699 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 17763
I20260620 19:08:31.816346 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:42011
--local_ip_for_outbound_sockets=127.1.154.131
--tserver_master_addrs=127.1.154.190:35761
--webserver_port=44341
--webserver_interface=127.1.154.131
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:08:31 dist-test-slave-6wrf krb5kdc[16991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982511, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:08:31.822631 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:31.807997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:38141 (local address 127.1.154.190:35761)
0620 19:08:31.808298 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:31.808303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:31.810064 (+ 1761us) server_negotiation.cc:407] Connection header received
0620 19:08:31.811092 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:31.811095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:31.811157 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:31.811263 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:31.812570 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.813116 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.814393 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.814627 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.814736 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:31.817738 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:31.817741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:31.817743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:31.817766 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:31.819564 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:31.820293 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:31.820299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:31.820301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:31.820458 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:31.821019 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:31.821023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:31.821025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:31.821337 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:31.821529 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:31.822242 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 19:08:31.822426 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260620 19:08:31.823817 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:31.808594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:38141)
0620 19:08:31.809857 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:31.809924 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:31.810866 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:31.811455 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:31.811463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:31.811868 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:31.812393 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:31.812429 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.813263 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.813267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:31.814271 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:31.814281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.814780 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.814799 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:31.814956 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:31.815839 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:31.815884 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:31.817518 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:31.819680 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:31.819689 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:31.819708 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:31.820088 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:31.820651 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:31.820657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:31.820662 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:31.820861 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:31.821484 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:31.821493 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:31.821919 (+ 426us) client_negotiation.cc:769] Sending connection context
0620 19:08:31.822159 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 19:08:31.822561 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"thread_start_us":679,"threads_started":1}
I20260620 19:08:31.825520 17899 heartbeater.cc:347] Connected to a master server at 127.1.154.190:35761
I20260620 19:08:31.825901 17899 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:31.826601 17899 heartbeater.cc:511] Master 127.1.154.190:35761 requested a full tablet report, sending...
I20260620 19:08:31.828143 17569 ts_manager.cc:194] Registered new tserver with Master: 0c8f9d494ae24bf3b2d1e1ba50d1755f (127.1.154.130:33703)
I20260620 19:08:31.828877 17569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:38141
W20260620 19:08:31.932114 17902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:31.932446 17902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:31.932519 17902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:31.936779 17902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:31.936887 17902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:31.936982 17902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:08:31.942685 17902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:42011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=44341
--tserver_master_addrs=127.1.154.190:35761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:08:31.943842 17902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:31.944919 17902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:31.956084 17907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:31.956374 17908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:31.956583 17902 server_base.cc:1034] running on GCE node
W20260620 19:08:31.956883 17910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:31.957494 17902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:31.958703 17902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:31.959939 17902 hybrid_clock.cc:648] HybridClock initialized: now 1781982511959915 us; error 47 us; skew 500 ppm
Jun 20 19:08:31 dist-test-slave-6wrf krb5kdc[16991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982511, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:31.963735 17902 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:08:31.964835 17902 webserver.cc:460] Webserver started at http://127.1.154.131:44341/ using document root <none> and password file <none>
I20260620 19:08:31.965462 17902 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:31.965582 17902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:31.970358 17902 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 19:08:31.973384 17917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:31.975150 17902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:08:31.975320 17902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "f8e6745d51754d8996377f5394072585"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-6wrf"
server_key: "561f4f9a797a3be5f6c0944e53c60e78"
server_key_iv: "926a5acba1b6e06f52065993e712719d"
server_key_version: "encryptionkey@0"
I20260620 19:08:31.975910 17902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:32.004536 17902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:32.007993 17902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:32.008244 17902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:32.008929 17902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:32.009837 17902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:32.009912 17902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:32.009980 17902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:32.010027 17902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:32.023205 17902 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:42011
I20260620 19:08:32.023222 18030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:42011 every 8 connection(s)
I20260620 19:08:32.024227 17902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:08:32.025418 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 17902
Jun 20 19:08:32 dist-test-slave-6wrf krb5kdc[16991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982511, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:08:32.036628 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.026234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58156 (local address 127.1.154.190:35761)
0620 19:08:32.026633 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:32.026638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:32.026654 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:08:32.026707 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:32.026710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:32.026761 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:32.026872 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:32.027872 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.028454 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.029691 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.029946 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.030035 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.032098 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:32.032102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:32.032104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:32.032127 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:32.034419 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:32.034878 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:32.034882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:32.034884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:32.034979 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:32.035520 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:32.035554 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:32.035558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:32.035823 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:32.036043 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:32.036251 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 19:08:32.036396 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":331,"thread_start_us":85,"threads_started":1}
I20260620 19:08:32.040788 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.026228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:50485 (local address 127.1.154.190:35761)
0620 19:08:32.026352 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:32.026357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:32.027841 (+ 1484us) server_negotiation.cc:407] Connection header received
0620 19:08:32.028765 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:32.028774 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:32.028816 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:32.028880 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:32.030304 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.030956 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.033048 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.033446 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.033564 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.036463 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:32.036467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:32.036470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:32.036491 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:32.038425 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:32.039039 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:32.039042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:32.039043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:32.039113 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:32.039471 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:32.039477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:32.039478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:32.039753 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:32.039843 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:32.040572 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 19:08:32.040674 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 19:08:32.041721 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.026804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:50485)
0620 19:08:32.027662 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:32.027718 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:32.028546 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:32.029087 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:32.029095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:32.029555 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:32.030164 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.030182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.031099 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.031102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.032818 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.032850 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.033637 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.033644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.033908 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.034521 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:32.034543 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:32.036269 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:32.038568 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:32.038575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:32.038592 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:32.038926 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:32.039179 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:32.039181 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:32.039182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:32.039316 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:32.039930 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:32.039939 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:32.040296 (+ 357us) client_negotiation.cc:769] Sending connection context
0620 19:08:32.040493 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 19:08:32.040755 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":141,"threads_started":1}
I20260620 19:08:32.043023 18031 heartbeater.cc:347] Connected to a master server at 127.1.154.190:35761
I20260620 19:08:32.043242 18031 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:32.043684 18031 heartbeater.cc:511] Master 127.1.154.190:35761 requested a full tablet report, sending...
I20260620 19:08:32.044864 17569 ts_manager.cc:194] Registered new tserver with Master: f8e6745d51754d8996377f5394072585 (127.1.154.131:42011)
I20260620 19:08:32.045431 17569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:50485
I20260620 19:08:32.051581 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:08:32.056584 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.052263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.1.154.190:35761)
0620 19:08:32.052406 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:32.052411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:32.052660 (+ 249us) server_negotiation.cc:407] Connection header received
0620 19:08:32.052761 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:32.052763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:32.052891 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:32.052986 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:08:32.053457 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.054102 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.055239 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.055505 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.055640 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.056044 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:08:32.056191 (+ 147us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 19:08:32.056447 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58166: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 19:08:32.056682 17764 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58166: token expired
W20260620 19:08:32.057161 17499 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.154.190:35761)
I20260620 19:08:32.065326 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.057576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.1.154.190:35761)
0620 19:08:32.057699 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:32.057702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:32.057716 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:08:32.057790 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:32.057792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:32.057828 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:32.057895 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:32.058570 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.059082 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.059918 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.060129 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.060239 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.061986 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:32.061989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:32.061991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:32.062014 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:32.063508 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:32.063983 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:32.063988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:32.063989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:32.064145 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:32.064518 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:32.064530 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:32.064532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:32.064774 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:32.064892 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:32.065109 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 19:08:32.065203 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:08:32.068859 17569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:08:32.072532 17569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:08:32.081645 17695 tablet_service.cc:1467] Processing CreateTablet for tablet 11390e618a564a6196c7d6d594e1a9fd (DEFAULT_TABLE table=test-table [id=ad2d8946019d421a884b2a2463250140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:32.082293 17695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11390e618a564a6196c7d6d594e1a9fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:32.085328 17653 tablet_bootstrap.cc:492] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a: Bootstrap starting.
I20260620 19:08:32.085762 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.080666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:33703 (local address 127.0.0.1:40436)
0620 19:08:32.080958 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:32.080973 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:32.081057 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:32.081709 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:32.081712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:32.081730 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:32.081843 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.081850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.083360 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.083364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.084391 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.084400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.085304 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.085308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.085415 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.085467 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 19:08:32.085522 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 19:08:32.085624 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":126,"threads_started":1}
I20260620 19:08:32.086169 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.080372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:42011 (local address 127.0.0.1:53206)
0620 19:08:32.080514 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:32.080530 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:32.080615 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:32.081632 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:32.081636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:32.081656 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:32.081789 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.081797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.083248 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.083251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.084494 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.084505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.085744 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.085748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.085860 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.085891 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 19:08:32.085953 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:08:32.086024 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":67}
I20260620 19:08:32.086349 17653 tablet_bootstrap.cc:654] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:32.086808 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.080531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53206 (local address 127.1.154.131:42011)
0620 19:08:32.081058 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:32.081070 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:32.081093 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:08:32.081180 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:32.081186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:32.081428 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:32.081621 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:32.081934 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.083120 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.084637 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.085545 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.085746 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.086541 (+ 795us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:32.086595 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 19:08:32.086700 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":111,"threads_started":1}
I20260620 19:08:32.087212 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.080979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.1.154.130:33703)
0620 19:08:32.081274 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:32.081283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:32.081302 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:08:32.081366 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:32.081370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:32.081537 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:32.082007 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:32.082173 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.083222 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.084530 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.085135 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.086606 (+ 1471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.086839 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:32.086890 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 19:08:32.086979 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":114,"threads_started":1}
I20260620 19:08:32.087913 17653 tablet_bootstrap.cc:492] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a: No bootstrap required, opened a new log
I20260620 19:08:32.088025 17653 ts_tablet_manager.cc:1397] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:08:32.088554 17653 raft_consensus.cc:348] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } }
I20260620 19:08:32.088699 17653 raft_consensus.cc:374] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:32.088750 17653 raft_consensus.cc:729] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e9dea926bd4ec69d12606ca66f348a, State: Initialized, Role: FOLLOWER
I20260620 19:08:32.088893 17653 consensus_queue.cc:260] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } }
I20260620 19:08:32.089216 17653 ts_tablet_manager.cc:1428] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:08:32.089735 17965 tablet_service.cc:1467] Processing CreateTablet for tablet 11390e618a564a6196c7d6d594e1a9fd (DEFAULT_TABLE table=test-table [id=ad2d8946019d421a884b2a2463250140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:32.089897 17833 tablet_service.cc:1467] Processing CreateTablet for tablet 11390e618a564a6196c7d6d594e1a9fd (DEFAULT_TABLE table=test-table [id=ad2d8946019d421a884b2a2463250140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:32.090720 17965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11390e618a564a6196c7d6d594e1a9fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:32.090819 17833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11390e618a564a6196c7d6d594e1a9fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:32.097944 18044 tablet_bootstrap.cc:492] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Bootstrap starting.
I20260620 19:08:32.098443 18045 tablet_bootstrap.cc:492] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585: Bootstrap starting.
I20260620 19:08:32.100838 18044 tablet_bootstrap.cc:654] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:32.100843 18045 tablet_bootstrap.cc:654] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:32.101576 18044 log.cc:824] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:32.101576 18045 log.cc:824] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:32.103480 18044 tablet_bootstrap.cc:492] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f: No bootstrap required, opened a new log
I20260620 19:08:32.103497 18045 tablet_bootstrap.cc:492] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585: No bootstrap required, opened a new log
I20260620 19:08:32.103688 18045 ts_tablet_manager.cc:1397] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 19:08:32.103700 18044 ts_tablet_manager.cc:1397] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 19:08:32.106734 18044 raft_consensus.cc:348] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } }
I20260620 19:08:32.106771 18045 raft_consensus.cc:348] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } }
I20260620 19:08:32.107008 18044 raft_consensus.cc:374] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:32.107002 18045 raft_consensus.cc:374] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:32.107097 18045 raft_consensus.cc:729] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e6745d51754d8996377f5394072585, State: Initialized, Role: FOLLOWER
I20260620 19:08:32.107093 18044 raft_consensus.cc:729] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8f9d494ae24bf3b2d1e1ba50d1755f, State: Initialized, Role: FOLLOWER
I20260620 19:08:32.107563 18045 consensus_queue.cc:260] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } }
I20260620 19:08:32.107563 18044 consensus_queue.cc:260] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } }
I20260620 19:08:32.108285 18031 heartbeater.cc:503] Master 127.1.154.190:35761 was elected leader, sending a full tablet report...
I20260620 19:08:32.108381 17899 heartbeater.cc:503] Master 127.1.154.190:35761 was elected leader, sending a full tablet report...
I20260620 19:08:32.108572 18045 ts_tablet_manager.cc:1428] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 19:08:32.108577 18044 ts_tablet_manager.cc:1428] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260620 19:08:32.168475 17767 raft_consensus.cc:482] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:32.168808 17767 raft_consensus.cc:504] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } }
I20260620 19:08:32.170677 17767 leader_election.cc:288] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e6745d51754d8996377f5394072585 (127.1.154.131:42011), 0c8f9d494ae24bf3b2d1e1ba50d1755f (127.1.154.130:33703)
I20260620 19:08:32.175596 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.171019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:33703 (local address 127.1.154.129:39835)
0620 19:08:32.171563 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:32.171578 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:32.171763 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:32.172108 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:32.172111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:32.172131 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:32.172275 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.172281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.173160 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.173166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.174245 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.174255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.175017 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.175021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.175124 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.175183 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 19:08:32.175284 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 19:08:32.175391 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":157,"thread_start_us":54,"threads_started":1}
I20260620 19:08:32.175575 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.171207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:39835 (local address 127.1.154.130:33703)
0620 19:08:32.171389 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:32.171395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:32.171687 (+ 292us) server_negotiation.cc:407] Connection header received
0620 19:08:32.171927 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:32.171931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:32.171996 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:32.172095 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:32.172403 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.173026 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.174388 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.174864 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.175017 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.175086 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:32.175300 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 19:08:32.175385 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 19:08:32.175801 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.171303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:45179 (local address 127.1.154.131:42011)
0620 19:08:32.171555 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:32.171561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:32.171720 (+ 159us) server_negotiation.cc:407] Connection header received
0620 19:08:32.172072 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:32.172145 (+ 73us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:32.172205 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:32.172298 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:32.172718 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.173247 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.174441 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.174914 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.175051 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.175265 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:32.175529 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 19:08:32.175633 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:08:32.175828 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.171019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:42011 (local address 127.1.154.129:45179)
0620 19:08:32.171555 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:32.171584 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:32.171782 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:32.172310 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:32.172314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:32.172333 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:32.172459 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.172465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.173369 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.173376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.174303 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.174314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.175039 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.175043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.175341 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.175382 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 19:08:32.175503 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 19:08:32.175620 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":4,"thread_start_us":128,"threads_started":1}
I20260620 19:08:32.176977 17985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11390e618a564a6196c7d6d594e1a9fd" candidate_uuid: "f4e9dea926bd4ec69d12606ca66f348a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e6745d51754d8996377f5394072585" is_pre_election: true
I20260620 19:08:32.177006 17853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11390e618a564a6196c7d6d594e1a9fd" candidate_uuid: "f4e9dea926bd4ec69d12606ca66f348a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" is_pre_election: true
I20260620 19:08:32.177316 17853 raft_consensus.cc:2454] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4e9dea926bd4ec69d12606ca66f348a in term 0.
I20260620 19:08:32.177320 17985 raft_consensus.cc:2454] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4e9dea926bd4ec69d12606ca66f348a in term 0.
I20260620 19:08:32.178017 17650 leader_election.cc:302] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f4e9dea926bd4ec69d12606ca66f348a, f8e6745d51754d8996377f5394072585; no voters:
I20260620 19:08:32.178366 17767 raft_consensus.cc:2781] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:08:32.178467 17767 raft_consensus.cc:482] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:08:32.178526 17767 raft_consensus.cc:3037] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:32.179875 17767 raft_consensus.cc:504] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } }
I20260620 19:08:32.180276 17767 leader_election.cc:288] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 1 election: Requested vote from peers f8e6745d51754d8996377f5394072585 (127.1.154.131:42011), 0c8f9d494ae24bf3b2d1e1ba50d1755f (127.1.154.130:33703)
I20260620 19:08:32.180714 17985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11390e618a564a6196c7d6d594e1a9fd" candidate_uuid: "f4e9dea926bd4ec69d12606ca66f348a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e6745d51754d8996377f5394072585"
I20260620 19:08:32.180876 17985 raft_consensus.cc:3037] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:32.181106 17853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11390e618a564a6196c7d6d594e1a9fd" candidate_uuid: "f4e9dea926bd4ec69d12606ca66f348a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f"
I20260620 19:08:32.181280 17853 raft_consensus.cc:3037] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:32.182214 17985 raft_consensus.cc:2454] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4e9dea926bd4ec69d12606ca66f348a in term 1.
I20260620 19:08:32.182667 17650 leader_election.cc:302] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f4e9dea926bd4ec69d12606ca66f348a, f8e6745d51754d8996377f5394072585; no voters:
I20260620 19:08:32.182999 17767 raft_consensus.cc:2781] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:32.183176 17853 raft_consensus.cc:2454] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4e9dea926bd4ec69d12606ca66f348a in term 1.
I20260620 19:08:32.183261 17767 raft_consensus.cc:686] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 LEADER]: Becoming Leader. State: Replica: f4e9dea926bd4ec69d12606ca66f348a, State: Running, Role: LEADER
I20260620 19:08:32.183562 17767 consensus_queue.cc:237] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } }
I20260620 19:08:32.186678 17569 catalog_manager.cc:5495] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a reported cstate change: term changed from 0 to 1, leader changed from <none> to f4e9dea926bd4ec69d12606ca66f348a (127.1.154.129). New cstate: current_term: 1 leader_uuid: "f4e9dea926bd4ec69d12606ca66f348a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } health_report { overall_health: HEALTHY } } }
I20260620 19:08:32.220198 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.216486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39360 (local address 127.1.154.129:38577)
0620 19:08:32.216793 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:32.216801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:32.216812 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:08:32.216887 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:32.216889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:32.216943 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:32.217072 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:08:32.217324 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.217869 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.218709 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.218938 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.219048 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.219340 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:08:32.219785 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:08:32.219910 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:32.219979 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 19:08:32.220035 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":101,"threads_started":1}
I20260620 19:08:32.224488 17853 raft_consensus.cc:1261] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 1 FOLLOWER]: Refusing update from remote peer f4e9dea926bd4ec69d12606ca66f348a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:32.224488 17985 raft_consensus.cc:1261] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585 [term 1 FOLLOWER]: Refusing update from remote peer f4e9dea926bd4ec69d12606ca66f348a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:32.225432 17767 consensus_queue.cc:1035] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:32.225718 18052 consensus_queue.cc:1035] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:32.232669 18056 mvcc.cc:205] Tried to move back new op lower bound from 7299000370064543744 to 7299000369906429952. Current Snapshot: MvccSnapshot[applied={T|T < 7299000370064543744}]
I20260620 19:08:32.233390 18057 mvcc.cc:205] Tried to move back new op lower bound from 7299000370064543744 to 7299000369906429952. Current Snapshot: MvccSnapshot[applied={T|T < 7299000370064543744}]
I20260620 19:08:32.238478 18059 mvcc.cc:205] Tried to move back new op lower bound from 7299000370064543744 to 7299000369906429952. Current Snapshot: MvccSnapshot[applied={T|T < 7299000370064543744 or (T in {7299000370064543744})}]
I20260620 19:08:32.242187 17569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:32.242429 17569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:32.245814 17569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Sending DeleteTablet for 3 replicas of tablet 11390e618a564a6196c7d6d594e1a9fd
I20260620 19:08:32.246654 17965 tablet_service.cc:1514] Processing DeleteTablet for tablet 11390e618a564a6196c7d6d594e1a9fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:32 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:53206
I20260620 19:08:32.246804 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 17611
I20260620 19:08:32.246744 17833 tablet_service.cc:1514] Processing DeleteTablet for tablet 11390e618a564a6196c7d6d594e1a9fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:32 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:40436
I20260620 19:08:32.249886 18067 tablet_replica.cc:330] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f: stopping tablet replica
I20260620 19:08:32.250201 18066 tablet_replica.cc:330] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585: stopping tablet replica
I20260620 19:08:32.250324 18067 raft_consensus.cc:2229] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:32.250562 18066 raft_consensus.cc:2229] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:32.250725 18067 raft_consensus.cc:2258] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:32.250864 18066 raft_consensus.cc:2258] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:32.252254 18066 ts_tablet_manager.cc:1905] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:32.252417 18067 ts_tablet_manager.cc:1905] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:08:32.254727 17558 connection.cc:535] client connection to 127.1.154.129:38577 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 19:08:32.254897 17558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:08:32.255007 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 17763
I20260620 19:08:32.256203 18066 ts_tablet_manager.cc:1918] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:32.256388 18066 log.cc:1196] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/11390e618a564a6196c7d6d594e1a9fd
I20260620 19:08:32.256744 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.256355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:38577 (local address 127.0.0.1:39362)
0620 19:08:32.256574 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:32.256646 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:38577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260620 19:08:32.256832 18066 ts_tablet_manager.cc:1939] T 11390e618a564a6196c7d6d594e1a9fd P f8e6745d51754d8996377f5394072585: Deleting consensus metadata
W20260620 19:08:32.257025 17558 catalog_manager.cc:4584] TS f4e9dea926bd4ec69d12606ca66f348a (127.1.154.129:38577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11390e618a564a6196c7d6d594e1a9fd: Network error: Client connection negotiation failed: client connection to 127.1.154.129:38577: connect: Connection refused (error 111)
I20260620 19:08:32.258242 17560 catalog_manager.cc:4841] TS f8e6745d51754d8996377f5394072585 (127.1.154.131:42011): tablet 11390e618a564a6196c7d6d594e1a9fd (table test-table [id=ad2d8946019d421a884b2a2463250140]) successfully deleted
I20260620 19:08:32.262693 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 17902
I20260620 19:08:32.264242 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.262824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:33703 (local address 127.0.0.1:40446)
0620 19:08:32.264058 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:32.264134 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.130:33703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1130}
I20260620 19:08:32.268663 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 17540
I20260620 19:08:52.276078 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:35761
--webserver_interface=127.1.154.190
--webserver_port=35601
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:35761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:52.395745 18069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:52.396018 18069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:52.396061 18069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:52.400173 18069 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:08:52.400234 18069 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:08:52.400251 18069 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:08:52.400265 18069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:52.400285 18069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:52.400302 18069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:52.400318 18069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:52.400384 18069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:52.405941 18069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:35761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:35761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=35601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:08:52.407027 18069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:52.408069 18069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:52.417088 18077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:52.417061 18074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:52.417096 18075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:52.417562 18069 server_base.cc:1034] running on GCE node
I20260620 19:08:52.418207 18069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:52.419446 18069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:52.420735 18069 hybrid_clock.cc:648] HybridClock initialized: now 1781982532420693 us; error 31 us; skew 500 ppm
Jun 20 19:08:52 dist-test-slave-6wrf krb5kdc[16991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982532, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:52.428303 18069 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:08:52.429411 18069 webserver.cc:460] Webserver started at http://127.1.154.190:35601/ using document root <none> and password file <none>
I20260620 19:08:52.430054 18069 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:52.430123 18069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:52.434883 18069 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:08:52.437745 18084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:52.439262 18069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 19:08:52.439433 18069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "1fbc3766918b45ceb67294eb0ae91f3d"
format_stamp: "Formatted at 2026-06-20 19:08:09 on dist-test-slave-6wrf"
server_key: "6eee58b40b2e04583d146084f7691338"
server_key_iv: "f86725cc54471aa6e64050bc64293ba8"
server_key_version: "encryptionkey@0"
I20260620 19:08:52.440028 18069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:52.457633 18069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:52.462298 18069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:52.462534 18069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:52.472280 18069 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:35761
I20260620 19:08:52.472604 18136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:35761 every 8 connection(s)
I20260620 19:08:52.473446 18069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:52.475073 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 18069
I20260620 19:08:52.475752 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:38577
--local_ip_for_outbound_sockets=127.1.154.129
--tserver_master_addrs=127.1.154.190:35761
--webserver_port=42399
--webserver_interface=127.1.154.129
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:08:52.479774 18137 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:08:52.481719 18137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Bootstrap starting.
I20260620 19:08:52.502202 18137 log.cc:824] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:52.514602 18137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:08:52.515017 18137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: Bootstrap complete.
I20260620 19:08:52.519433 18137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } }
I20260620 19:08:52.519860 18137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fbc3766918b45ceb67294eb0ae91f3d, State: Initialized, Role: FOLLOWER
I20260620 19:08:52.520408 18137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } }
I20260620 19:08:52.520550 18137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:52.520642 18137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:52.520766 18137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:08:52.522922 18137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } }
I20260620 19:08:52.523274 18137 leader_election.cc:302] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fbc3766918b45ceb67294eb0ae91f3d; no voters:
I20260620 19:08:52.523643 18137 leader_election.cc:288] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 19:08:52.523743 18141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:08:52.523978 18141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [term 3 LEADER]: Becoming Leader. State: Replica: 1fbc3766918b45ceb67294eb0ae91f3d, State: Running, Role: LEADER
I20260620 19:08:52.524345 18141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } }
I20260620 19:08:52.524927 18137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:52.525823 18143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fbc3766918b45ceb67294eb0ae91f3d. Latest consensus state: current_term: 3 leader_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } } }
I20260620 19:08:52.525943 18143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:52.525882 18142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbc3766918b45ceb67294eb0ae91f3d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 35761 } } }
I20260620 19:08:52.526005 18142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:52.530436 18157 catalog_manager.cc:1259] Loaded cluster ID: fb6d796b51a7402e9ef44d36bfc79661
I20260620 19:08:52.530514 18157 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: loading cluster ID for follower catalog manager: success
I20260620 19:08:52.533185 18157 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1fbc3766918b45ceb67294eb0ae91f3d: acquiring CA information for follower catalog manager: success
F20260620 19:08:52.535394 18157 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc6484069ad google::LogMessage::Fail() at ??:0
@ 0x7fc64840988a google::LogMessage::SendToLog() at ??:0
@ 0x7fc6484064de google::LogMessage::Flush() at ??:0
@ 0x7fc648407a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc64b0ca18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc652d6b989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fc652d48023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fc652d4854b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fc652d41652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fc652d409e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7fc652d87541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc64a649512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc64bc8d6db start_thread at ??:0
@ 0x7fc646f0461f clone at ??:0
W20260620 19:08:52.623358 18139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:52.623768 18139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:52.623840 18139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:52.630983 18139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:52.631089 18139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:52.631196 18139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:08:52.639911 18139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:38577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=42399
--tserver_master_addrs=127.1.154.190:35761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:08:52.641471 18139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:52.642851 18139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:52.654763 18180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:52.654775 18177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:52.654765 18178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:52.655607 18139 server_base.cc:1034] running on GCE node
I20260620 19:08:52.656108 18139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:52.656909 18139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:52.658107 18139 hybrid_clock.cc:648] HybridClock initialized: now 1781982532658070 us; error 43 us; skew 500 ppm
Jun 20 19:08:52 dist-test-slave-6wrf krb5kdc[16991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982532, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:52.661827 18139 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:08:52.662792 18139 webserver.cc:460] Webserver started at http://127.1.154.129:42399/ using document root <none> and password file <none>
I20260620 19:08:52.663403 18139 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:52.663470 18139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:52.667918 18139 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:08:52.670418 18187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:52.672431 18139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:08:52.672700 18139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "f4e9dea926bd4ec69d12606ca66f348a"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-6wrf"
server_key: "d9ddadce4a69e31f1a3d294236392abc"
server_key_iv: "a44f78c5d7ad63617059b1a57f5fc2c5"
server_key_version: "encryptionkey@0"
I20260620 19:08:52.673771 18139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:52.697005 18139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:52.701781 18139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:52.702067 18139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:52.702931 18139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:52.704531 18194 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:08:52.710192 18139 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:08:52.710310 18139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 19:08:52.710394 18139 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:08:52.712167 18139 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:08:52.712244 18139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 19:08:52.712548 18194 tablet_bootstrap.cc:492] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a: Bootstrap starting.
I20260620 19:08:52.722991 18139 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:38577
I20260620 19:08:52.723419 18301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:38577 every 8 connection(s)
I20260620 19:08:52.724260 18139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:08:52.725278 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 18139
I20260620 19:08:52.725765 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:33703
--local_ip_for_outbound_sockets=127.1.154.130
--tserver_master_addrs=127.1.154.190:35761
--webserver_port=39725
--webserver_interface=127.1.154.130
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:08:52.735093 18194 log.cc:824] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:52.744557 18194 tablet_bootstrap.cc:492] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:08:52.744994 18194 tablet_bootstrap.cc:492] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a: Bootstrap complete.
I20260620 19:08:52.746475 18194 ts_tablet_manager.cc:1397] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a: Time spent bootstrapping tablet: real 0.034s user 0.029s sys 0.004s
I20260620 19:08:52.749871 18194 raft_consensus.cc:348] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } }
I20260620 19:08:52.750416 18194 raft_consensus.cc:729] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e9dea926bd4ec69d12606ca66f348a, State: Initialized, Role: FOLLOWER
I20260620 19:08:52.750936 18194 consensus_queue.cc:260] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } }
I20260620 19:08:52.751974 18194 ts_tablet_manager.cc:1428] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 19:08:52.815886 18304 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:52.726503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:52829)
0620 19:08:52.727046 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:52.727102 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:52.728274 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:52.814765 (+ 86491us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":138,"threads_started":1}
W20260620 19:08:52.816707 18302 heartbeater.cc:644] Failed to heartbeat to 127.1.154.190:35761 (0 consecutive failures): Network error: Failed to ping master at 127.1.154.190:35761: Client connection negotiation failed: client connection to 127.1.154.190:35761: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:08:52.817544 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:52.817266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:35151)
0620 19:08:52.817401 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:52.817477 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 19:08:52.818313 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:52.818081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:44845)
0620 19:08:52.818215 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:52.818263 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 19:08:52.818588 18302 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:08:52.858414 18305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:52.858700 18305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:52.858747 18305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:52.863106 18305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:52.863183 18305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:52.863274 18305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:08:52.868857 18305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:33703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=39725
--tserver_master_addrs=127.1.154.190:35761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:08:52.870116 18305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:52.871155 18305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:08:52.881978 18305 server_base.cc:1034] running on GCE node
W20260620 19:08:52.881860 18313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:52.881870 18312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:52.881892 18315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:52.882659 18305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:52.883381 18305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:52.884558 18305 hybrid_clock.cc:648] HybridClock initialized: now 1781982532884541 us; error 29 us; skew 500 ppm
Jun 20 19:08:52 dist-test-slave-6wrf krb5kdc[16991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982532, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:52.887523 18305 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:08:52.888495 18305 webserver.cc:460] Webserver started at http://127.1.154.130:39725/ using document root <none> and password file <none>
I20260620 19:08:52.889073 18305 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:52.889140 18305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:52.893826 18305 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:08:52.897065 18322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:52.898699 18305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:08:52.898823 18305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-6wrf"
server_key: "23c4325a380fdb3835fe43b33e57e841"
server_key_iv: "4d3212760d2270f23f879a7c00eed645"
server_key_version: "encryptionkey@0"
I20260620 19:08:52.899433 18305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:52.915277 18305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:52.919449 18305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:52.919725 18305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:52.920364 18305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:52.921546 18329 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 19:08:52.925360 18329 ts_tablet_manager.cc:1852] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 19:08:52.925865 18329 ts_tablet_manager.cc:1905] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:52.930116 18329 ts_tablet_manager.cc:1918] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:52.930347 18329 log.cc:1196] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/11390e618a564a6196c7d6d594e1a9fd
I20260620 19:08:52.930783 18329 ts_tablet_manager.cc:1939] T 11390e618a564a6196c7d6d594e1a9fd P 0c8f9d494ae24bf3b2d1e1ba50d1755f: Deleting consensus metadata
I20260620 19:08:52.931210 18305 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 19:08:52.931298 18305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260620 19:08:52.931401 18305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:52.931449 18305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:52.942823 18305 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:33703
I20260620 19:08:52.942989 18436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:33703 every 8 connection(s)
I20260620 19:08:52.944252 18305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:08:52.945169 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 18305
I20260620 19:08:52.945715 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:42011
--local_ip_for_outbound_sockets=127.1.154.131
--tserver_master_addrs=127.1.154.190:35761
--webserver_port=44341
--webserver_interface=127.1.154.131
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:08:52.948808 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:52.947429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:57539)
0620 19:08:52.947928 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:52.948058 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":153,"threads_started":1}
W20260620 19:08:52.949719 18437 heartbeater.cc:644] Failed to heartbeat to 127.1.154.190:35761 (0 consecutive failures): Network error: Failed to ping master at 127.1.154.190:35761: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
I20260620 19:08:52.950414 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:52.950171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:42407)
0620 19:08:52.950296 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:52.950348 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 19:08:52.951144 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:52.950957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:51703)
0620 19:08:52.951058 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:52.951098 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 19:08:52.951370 18437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:08:53.063791 18439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:53.064128 18439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:53.064179 18439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:53.069020 18439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:53.069113 18439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:53.069231 18439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:08:53.075547 18439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:42011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=44341
--tserver_master_addrs=127.1.154.190:35761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:08:53.076656 18439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:53.077798 18439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:53.087998 18448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:53.088110 18439 server_base.cc:1034] running on GCE node
W20260620 19:08:53.088052 18445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:53.088038 18446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:53.088730 18439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:53.089397 18439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:53.090577 18439 hybrid_clock.cc:648] HybridClock initialized: now 1781982533090555 us; error 50 us; skew 500 ppm
Jun 20 19:08:53 dist-test-slave-6wrf krb5kdc[16991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982533, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:53.093923 18439 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:08:53.095204 18439 webserver.cc:460] Webserver started at http://127.1.154.131:44341/ using document root <none> and password file <none>
I20260620 19:08:53.095949 18439 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:53.096048 18439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:53.100759 18439 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:08:53.103318 18455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:53.104803 18439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:08:53.104974 18439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "f8e6745d51754d8996377f5394072585"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-6wrf"
server_key: "561f4f9a797a3be5f6c0944e53c60e78"
server_key_iv: "926a5acba1b6e06f52065993e712719d"
server_key_version: "encryptionkey@0"
I20260620 19:08:53.106056 18439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:53.121097 18439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:53.124809 18439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:53.125049 18439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:53.125741 18439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:53.126570 18439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:53.126641 18439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:53.126715 18439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:53.126766 18439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:53.137022 18439 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:42011
I20260620 19:08:53.137053 18568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:42011 every 8 connection(s)
I20260620 19:08:53.138206 18439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:08:53.142452 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:53.140553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:58221)
0620 19:08:53.141068 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:53.141185 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":114,"threads_started":1}
W20260620 19:08:53.142966 18569 heartbeater.cc:644] Failed to heartbeat to 127.1.154.190:35761 (0 consecutive failures): Network error: Failed to ping master at 127.1.154.190:35761: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
I20260620 19:08:53.143082 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 18439
I20260620 19:08:53.143922 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:53.143631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:48785)
0620 19:08:53.143812 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:53.143866 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 19:08:53.144712 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:53.144451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:39941)
0620 19:08:53.144583 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:53.144655 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 19:08:53.144912 18569 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:08:53.819811 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:53.819323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:60485)
0620 19:08:53.819621 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:53.819698 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":84,"threads_started":1}
I20260620 19:08:53.952630 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:53.952001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:46517)
0620 19:08:53.952402 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:53.952497 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":79,"threads_started":1}
I20260620 19:08:54.146199 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:54.145542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:44521)
0620 19:08:54.145944 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:54.146038 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
I20260620 19:08:54.264930 18576 raft_consensus.cc:482] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:54.265183 18576 raft_consensus.cc:504] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } }
I20260620 19:08:54.266379 18576 leader_election.cc:288] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e6745d51754d8996377f5394072585 (127.1.154.131:42011), 0c8f9d494ae24bf3b2d1e1ba50d1755f (127.1.154.130:33703)
Jun 20 19:08:54 dist-test-slave-6wrf krb5kdc[16991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982532, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.130@KRBTEST.COM
Jun 20 19:08:54 dist-test-slave-6wrf krb5kdc[16991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982532, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.131@KRBTEST.COM
I20260620 19:08:54.283092 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:54.266611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:33703 (local address 127.1.154.129:36189)
0620 19:08:54.266785 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:54.266800 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:54.266907 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:54.269017 (+ 2110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:54.269019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:54.269444 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:54.269972 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:54.269986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:54.271270 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:54.271273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:54.272216 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:54.272225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:54.272926 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:54.272930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:54.273164 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:54.273940 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:54.273976 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:54.275787 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:54.279403 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:54.279410 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:54.279426 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:54.280288 (+ 862us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:54.281007 (+ 719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:54.281009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:54.281010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:54.281108 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:54.282180 (+ 1072us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:54.282182 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:54.282304 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 19:08:54.282497 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 19:08:54.282885 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":92,"spinlock_wait_cycles":22656}
I20260620 19:08:54.283278 18578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:54.266816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:36805 (local address 127.1.154.131:42011)
0620 19:08:54.267173 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:54.267194 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:54.267229 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:08:54.267991 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:54.267998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:54.268295 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:54.268542 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:54.270186 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:54.271122 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:54.272346 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:54.272798 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:54.272972 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:54.276350 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:54.276355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:54.276369 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:54.276407 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:54.279177 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:54.280401 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:54.280412 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:54.280421 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:54.280624 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:54.281220 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:54.281224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:54.281227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:54.281816 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:54.281935 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:54.282762 (+ 827us) server_negotiation.cc:299] Negotiation successful
0620 19:08:54.283056 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":63,"threads_started":1}
I20260620 19:08:54.283427 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:54.266839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:36189 (local address 127.1.154.130:33703)
0620 19:08:54.267245 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:54.267256 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:54.267285 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:08:54.267957 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:54.267963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:54.268571 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:54.269096 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:54.270149 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:54.271121 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:54.272344 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:54.272796 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:54.272976 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:54.276038 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:54.276045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:54.276072 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:54.276104 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:54.279177 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:54.280510 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:54.280516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:54.280521 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:54.280850 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:54.281369 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:54.281372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:54.281374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:54.282044 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:54.282211 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:54.282692 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 19:08:54.283215 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":96,"threads_started":1}
I20260620 19:08:54.283291 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:54.266613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:42011 (local address 127.1.154.129:36805)
0620 19:08:54.266984 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:54.266998 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:54.267099 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:54.268861 (+ 1762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:54.268872 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:54.269559 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:54.269972 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:54.269987 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:54.271269 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:54.271273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:54.272214 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:54.272223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:54.272926 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:54.272929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:54.273167 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:54.274005 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:54.274046 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:54.276150 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:54.279461 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:54.279464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:54.279465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:54.280233 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:54.280843 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:54.280846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:54.280848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:54.281093 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:54.281955 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:54.281965 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:54.282294 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 19:08:54.282605 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 19:08:54.283051 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":317,"spinlock_wait_cycles":14336,"thread_start_us":85,"threads_started":1}
W20260620 19:08:54.286664 18189 leader_election.cc:341] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8f9d494ae24bf3b2d1e1ba50d1755f (127.1.154.130:33703): Not found: Tablet not found: 11390e618a564a6196c7d6d594e1a9fd
W20260620 19:08:54.287065 18191 leader_election.cc:341] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8e6745d51754d8996377f5394072585 (127.1.154.131:42011): Not found: Tablet not found: 11390e618a564a6196c7d6d594e1a9fd
I20260620 19:08:54.287168 18191 leader_election.cc:302] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4e9dea926bd4ec69d12606ca66f348a; no voters: 0c8f9d494ae24bf3b2d1e1ba50d1755f, f8e6745d51754d8996377f5394072585
I20260620 19:08:54.287472 18576 raft_consensus.cc:2726] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:08:54.821556 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:54.820983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:58665)
0620 19:08:54.821357 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:54.821445 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":115,"threads_started":1}
I20260620 19:08:54.954532 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:54.953945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:35021)
0620 19:08:54.954324 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:54.954392 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":120,"threads_started":1}
I20260620 19:08:55.147709 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:55.147155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:54793)
0620 19:08:55.147493 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:55.147576 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":78,"threads_started":1}
I20260620 19:08:55.823640 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:55.822986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:38915)
0620 19:08:55.823423 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:55.823530 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":140,"threads_started":1}
I20260620 19:08:55.956147 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:55.955462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:50551)
0620 19:08:55.955898 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:55.956009 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":119,"threads_started":1}
I20260620 19:08:56.094607 18585 raft_consensus.cc:482] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:56.094786 18585 raft_consensus.cc:504] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } }
I20260620 19:08:56.095425 18585 leader_election.cc:288] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e6745d51754d8996377f5394072585 (127.1.154.131:42011), 0c8f9d494ae24bf3b2d1e1ba50d1755f (127.1.154.130:33703)
W20260620 19:08:56.096338 18191 leader_election.cc:341] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8e6745d51754d8996377f5394072585 (127.1.154.131:42011): Not found: Tablet not found: 11390e618a564a6196c7d6d594e1a9fd
W20260620 19:08:56.096658 18189 leader_election.cc:341] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8f9d494ae24bf3b2d1e1ba50d1755f (127.1.154.130:33703): Not found: Tablet not found: 11390e618a564a6196c7d6d594e1a9fd
I20260620 19:08:56.096798 18189 leader_election.cc:302] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4e9dea926bd4ec69d12606ca66f348a; no voters: 0c8f9d494ae24bf3b2d1e1ba50d1755f, f8e6745d51754d8996377f5394072585
I20260620 19:08:56.097051 18585 raft_consensus.cc:2726] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:08:56.149573 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:56.148839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:59789)
0620 19:08:56.149196 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:56.149268 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260620 19:08:56.825482 18587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:56.824909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:40735)
0620 19:08:56.825255 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:56.825344 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":89,"threads_started":1}
I20260620 19:08:56.957950 18588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:56.957237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:34883)
0620 19:08:56.957715 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:56.957798 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":99,"threads_started":1}
I20260620 19:08:57.151531 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:57.150819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:56553)
0620 19:08:57.151318 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:57.151389 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":305,"threads_started":1}
I20260620 19:08:57.827630 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:57.826960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:60535)
0620 19:08:57.827433 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:57.827519 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":108,"threads_started":1}
I20260620 19:08:57.959787 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:57.959142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:53367)
0620 19:08:57.959554 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:57.959653 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":99,"threads_started":1}
I20260620 19:08:58.152938 18592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:58.152473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:51163)
0620 19:08:58.152763 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:58.152828 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260620 19:08:58.829836 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:58.829057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:51817)
0620 19:08:58.829414 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:58.829699 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":144,"threads_started":1}
I20260620 19:08:58.961786 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:58.960953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:60267)
0620 19:08:58.961402 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:58.961546 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":88,"threads_started":1}
I20260620 19:08:59.154717 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:59.154106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:40139)
0620 19:08:59.154483 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:59.154562 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":134,"threads_started":1}
I20260620 19:08:59.214272 18596 raft_consensus.cc:482] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:59.214411 18596 raft_consensus.cc:504] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } }
I20260620 19:08:59.214900 18596 leader_election.cc:288] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e6745d51754d8996377f5394072585 (127.1.154.131:42011), 0c8f9d494ae24bf3b2d1e1ba50d1755f (127.1.154.130:33703)
W20260620 19:08:59.215879 18189 leader_election.cc:341] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8f9d494ae24bf3b2d1e1ba50d1755f (127.1.154.130:33703): Not found: Tablet not found: 11390e618a564a6196c7d6d594e1a9fd
W20260620 19:08:59.215970 18191 leader_election.cc:341] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8e6745d51754d8996377f5394072585 (127.1.154.131:42011): Not found: Tablet not found: 11390e618a564a6196c7d6d594e1a9fd
I20260620 19:08:59.216099 18191 leader_election.cc:302] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4e9dea926bd4ec69d12606ca66f348a; no voters: 0c8f9d494ae24bf3b2d1e1ba50d1755f, f8e6745d51754d8996377f5394072585
I20260620 19:08:59.216306 18596 raft_consensus.cc:2726] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:08:59.831547 18597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:59.830941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:39981)
0620 19:08:59.831377 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:59.831446 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":177,"threads_started":1}
I20260620 19:08:59.963790 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:59.963106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:50847)
0620 19:08:59.963592 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:59.963677 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":115,"threads_started":1}
I20260620 19:09:00.156543 18599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:00.155773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:40227)
0620 19:09:00.156273 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:00.156373 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":109,"threads_started":1}
I20260620 19:09:00.832823 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:00.832378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:36581)
0620 19:09:00.832670 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:00.832725 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":90,"threads_started":1}
I20260620 19:09:00.965178 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:00.964713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:44467)
0620 19:09:00.964995 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:00.965056 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260620 19:09:01.158347 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:01.157830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:41965)
0620 19:09:01.158158 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:01.158219 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":119,"threads_started":1}
I20260620 19:09:01.834368 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:01.833927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:48793)
0620 19:09:01.834203 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:01.834273 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260620 19:09:01.966799 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:01.966203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:41983)
0620 19:09:01.966561 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:01.966627 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":118,"threads_started":1}
I20260620 19:09:01.986032 18605 raft_consensus.cc:482] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:09:01.986233 18605 raft_consensus.cc:504] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e6745d51754d8996377f5394072585" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42011 } } peers { permanent_uuid: "0c8f9d494ae24bf3b2d1e1ba50d1755f" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 33703 } } peers { permanent_uuid: "f4e9dea926bd4ec69d12606ca66f348a" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 38577 } }
I20260620 19:09:01.987002 18605 leader_election.cc:288] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e6745d51754d8996377f5394072585 (127.1.154.131:42011), 0c8f9d494ae24bf3b2d1e1ba50d1755f (127.1.154.130:33703)
W20260620 19:09:01.988051 18189 leader_election.cc:341] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8f9d494ae24bf3b2d1e1ba50d1755f (127.1.154.130:33703): Not found: Tablet not found: 11390e618a564a6196c7d6d594e1a9fd
W20260620 19:09:01.988199 18191 leader_election.cc:341] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8e6745d51754d8996377f5394072585 (127.1.154.131:42011): Not found: Tablet not found: 11390e618a564a6196c7d6d594e1a9fd
I20260620 19:09:01.988345 18191 leader_election.cc:302] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4e9dea926bd4ec69d12606ca66f348a; no voters: 0c8f9d494ae24bf3b2d1e1ba50d1755f, f8e6745d51754d8996377f5394072585
I20260620 19:09:01.988588 18605 raft_consensus.cc:2726] T 11390e618a564a6196c7d6d594e1a9fd P f4e9dea926bd4ec69d12606ca66f348a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:09:02.159940 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:02.159373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:42169)
0620 19:09:02.159736 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:02.159821 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":99,"threads_started":1}
I20260620 19:09:02.835911 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:02.835479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:40513)
0620 19:09:02.835744 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:02.835817 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
I20260620 19:09:02.968786 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:02.968004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:37229)
0620 19:09:02.968583 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:02.968656 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":140,"threads_started":1}
I20260620 19:09:03.161640 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:03.160961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:40651)
0620 19:09:03.161388 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:03.161479 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":116,"threads_started":1}
I20260620 19:09:03.837720 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:03.836928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:46173)
0620 19:09:03.837383 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:03.837481 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":85,"threads_started":1}
I20260620 19:09:03.970757 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:03.970059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:57573)
0620 19:09:03.970518 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:03.970603 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260620 19:09:04.163270 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:04.162582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:33037)
0620 19:09:04.163074 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:04.163143 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":101,"threads_started":1}
I20260620 19:09:04.839648 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:04.838934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:50129)
0620 19:09:04.839439 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:04.839518 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":143,"threads_started":1}
I20260620 19:09:04.972371 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:04.971734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:46823)
0620 19:09:04.972126 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:04.972227 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":81,"threads_started":1}
I20260620 19:09:05.165019 18615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:05.164427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:53817)
0620 19:09:05.164801 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:05.164885 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":106,"threads_started":1}
I20260620 19:09:05.841655 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:05.841022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:60887)
0620 19:09:05.841412 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:05.841489 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":136,"threads_started":1}
I20260620 19:09:05.974351 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:05.973471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:46443)
0620 19:09:05.974097 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:05.974214 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":84,"threads_started":1}
I20260620 19:09:06.166980 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:06.166358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:46289)
0620 19:09:06.166737 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:06.166832 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":96,"threads_started":1}
I20260620 19:09:06.843657 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:06.842971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:37463)
0620 19:09:06.843422 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:06.843497 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":82,"threads_started":1}
I20260620 19:09:06.976442 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:06.975821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:36635)
0620 19:09:06.976221 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:06.976306 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":112,"threads_started":1}
I20260620 19:09:07.168725 18621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:07.167941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.131:59003)
0620 19:09:07.168442 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:07.168574 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":88,"threads_started":1}
I20260620 19:09:07.845553 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:07.845093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.129:40207)
0620 19:09:07.845390 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:07.845454 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
I20260620 19:09:07.978051 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:07.977506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:35761 (local address 127.1.154.130:57429)
0620 19:09:07.977885 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:07.977946 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.190:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":153,"threads_started":1}
I20260620 19:09:08.150449 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 18139
I20260620 19:09:08.156715 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 18305
I20260620 19:09:08.163354 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 18439
2026-06-20T19:09:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59778 ms)
[----------] 30 tests from SecurityITest (254753 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:08 dist-test-slave-6wrf krb5kdc[18626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:08 dist-test-slave-6wrf krb5kdc[18626](info): set up 2 sockets
Jun 20 19:09:08 dist-test-slave-6wrf krb5kdc[18626](info): commencing operation
krb5kdc: starting...
W20260620 19:09:09.250520 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:09 dist-test-slave-6wrf krb5kdc[18626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:09:09.380992 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:38669
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:38669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:09.493461 18639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:09.493798 18639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:09.493842 18639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:09.497802 18639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:09.497864 18639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:09.497884 18639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:09.497900 18639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:09.497915 18639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:09.503224 18639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:38669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:38669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:09.504262 18639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:09.505249 18639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:09:09.514814 18639 server_base.cc:1034] running on GCE node
W20260620 19:09:09.514779 18645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:09.514918 18647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:09.514793 18644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:09.515661 18639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:09.516855 18639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:09.518066 18639 hybrid_clock.cc:648] HybridClock initialized: now 1781982549518039 us; error 39 us; skew 500 ppm
Jun 20 19:09:09 dist-test-slave-6wrf krb5kdc[18626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982549, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:09.520910 18639 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:09:09.521900 18639 webserver.cc:460] Webserver started at http://127.1.154.190:43811/ using document root <none> and password file <none>
I20260620 19:09:09.522482 18639 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:09.522569 18639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:09.522827 18639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:09.525092 18639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "d222434bb6684dd79391cc31975f8575"
format_stamp: "Formatted at 2026-06-20 19:09:09 on dist-test-slave-6wrf"
server_key: "61fe164080874164c4cd046994b56bb5"
server_key_iv: "bf386731712e3a3bcd9133532acbbbc1"
server_key_version: "encryptionkey@0"
I20260620 19:09:09.525766 18639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "d222434bb6684dd79391cc31975f8575"
format_stamp: "Formatted at 2026-06-20 19:09:09 on dist-test-slave-6wrf"
server_key: "61fe164080874164c4cd046994b56bb5"
server_key_iv: "bf386731712e3a3bcd9133532acbbbc1"
server_key_version: "encryptionkey@0"
I20260620 19:09:09.529583 18639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:09:09.532419 18654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:09.534132 18639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 19:09:09.534287 18639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "d222434bb6684dd79391cc31975f8575"
format_stamp: "Formatted at 2026-06-20 19:09:09 on dist-test-slave-6wrf"
server_key: "61fe164080874164c4cd046994b56bb5"
server_key_iv: "bf386731712e3a3bcd9133532acbbbc1"
server_key_version: "encryptionkey@0"
I20260620 19:09:09.534403 18639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:09.556993 18639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:09.560333 18639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:09.560549 18639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:09.568315 18639 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:38669
I20260620 19:09:09.568364 18706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:38669 every 8 connection(s)
I20260620 19:09:09.569407 18639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:09.572368 18707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:09.577838 18707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575: Bootstrap starting.
I20260620 19:09:09.578259 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 18639
I20260620 19:09:09.578408 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:09.578680 1642 external_mini_cluster.cc:1428] Setting key 4bd43c6aaaad6b4eeee72e43be9f419f
I20260620 19:09:09.580816 18707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:09.581562 18707 log.cc:824] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:09.583792 18707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575: No bootstrap required, opened a new log
I20260620 19:09:09.588675 18707 raft_consensus.cc:348] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d222434bb6684dd79391cc31975f8575" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38669 } }
I20260620 19:09:09.589138 18707 raft_consensus.cc:374] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:09.589208 18707 raft_consensus.cc:729] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d222434bb6684dd79391cc31975f8575, State: Initialized, Role: FOLLOWER
I20260620 19:09:09.590075 18707 consensus_queue.cc:260] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d222434bb6684dd79391cc31975f8575" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38669 } }
I20260620 19:09:09.590302 18707 raft_consensus.cc:388] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:09.590392 18707 raft_consensus.cc:482] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Jun 20 19:09:09 dist-test-slave-6wrf krb5kdc[18626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:09.590526 18707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:09.591856 18707 raft_consensus.cc:504] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d222434bb6684dd79391cc31975f8575" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38669 } }
I20260620 19:09:09.592309 18707 leader_election.cc:302] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d222434bb6684dd79391cc31975f8575; no voters:
I20260620 19:09:09.592732 18707 leader_election.cc:288] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:09.592979 18712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:09.593354 18712 raft_consensus.cc:686] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [term 1 LEADER]: Becoming Leader. State: Replica: d222434bb6684dd79391cc31975f8575, State: Running, Role: LEADER
I20260620 19:09:09.593947 18712 consensus_queue.cc:237] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d222434bb6684dd79391cc31975f8575" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38669 } }
I20260620 19:09:09.594882 18707 sys_catalog.cc:564] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:09.596449 18714 sys_catalog.cc:455] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d222434bb6684dd79391cc31975f8575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d222434bb6684dd79391cc31975f8575" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38669 } } }
I20260620 19:09:09.596623 18714 sys_catalog.cc:458] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:09.596943 18713 sys_catalog.cc:455] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d222434bb6684dd79391cc31975f8575. Latest consensus state: current_term: 1 leader_uuid: "d222434bb6684dd79391cc31975f8575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d222434bb6684dd79391cc31975f8575" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38669 } } }
I20260620 19:09:09.597070 18713 sys_catalog.cc:458] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:09.597344 18721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:09.598245 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:09.580305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.1.154.190:38669)
0620 19:09:09.580978 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:09.581003 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:09.581055 (+ 52us) server_negotiation.cc:407] Connection header received
0620 19:09:09.582330 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:09.582355 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:09.582708 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:09.583040 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:09.584131 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:09.585129 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:09.586538 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:09.587254 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:09.587456 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:09.591294 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:09.591301 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:09.591322 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:09.591374 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:09.594142 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:09.594771 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:09.594778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:09.594786 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:09.594945 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:09.595262 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:09.595265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:09.595266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:09.595809 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:09.595999 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:09.596850 (+ 851us) server_negotiation.cc:299] Negotiation successful
0620 19:09:09.597214 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":150,"threads_started":1}
I20260620 19:09:09.600631 18721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:09.606943 18721 catalog_manager.cc:1347] Generated new cluster ID: e771fe3322324f888cb48e2485a131f2
I20260620 19:09:09.607020 18721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:09.615770 18721 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:09.617930 18721 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:09.630218 18721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575: Generated new TSK 0
I20260620 19:09:09.630940 18721 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:09:09.665875 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:38669
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:09.779307 18733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:09.779588 18733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:09.779632 18733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:09.784329 18733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:09.784404 18733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:09.784466 18733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:09:09.791174 18733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:09.792790 18733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:09.794003 18733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:09.804579 18738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:09.804641 18739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:09.804698 18741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:09.804880 18733 server_base.cc:1034] running on GCE node
I20260620 19:09:09.805514 18733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:09.806243 18733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:09.807444 18733 hybrid_clock.cc:648] HybridClock initialized: now 1781982549807395 us; error 64 us; skew 500 ppm
Jun 20 19:09:09 dist-test-slave-6wrf krb5kdc[18626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982549, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:09.810899 18733 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:09:09.811760 18733 webserver.cc:460] Webserver started at http://127.1.154.129:37991/ using document root <none> and password file <none>
I20260620 19:09:09.812357 18733 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:09.812448 18733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:09.812672 18733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:09.814781 18733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6"
format_stamp: "Formatted at 2026-06-20 19:09:09 on dist-test-slave-6wrf"
server_key: "d35b9c99f1fa60efb2855fa95c591ce5"
server_key_iv: "aee20d4c1ff00043339e6ffa344bc2db"
server_key_version: "encryptionkey@0"
I20260620 19:09:09.815331 18733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6"
format_stamp: "Formatted at 2026-06-20 19:09:09 on dist-test-slave-6wrf"
server_key: "d35b9c99f1fa60efb2855fa95c591ce5"
server_key_iv: "aee20d4c1ff00043339e6ffa344bc2db"
server_key_version: "encryptionkey@0"
I20260620 19:09:09.819130 18733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:09:09.821656 18748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:09.822956 18733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:09:09.823105 18733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6"
format_stamp: "Formatted at 2026-06-20 19:09:09 on dist-test-slave-6wrf"
server_key: "d35b9c99f1fa60efb2855fa95c591ce5"
server_key_iv: "aee20d4c1ff00043339e6ffa344bc2db"
server_key_version: "encryptionkey@0"
I20260620 19:09:09.823216 18733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:09.851138 18733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:09.855985 18733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:09.856293 18733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:09.857147 18733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:09.858314 18733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:09.858398 18733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:09.858482 18733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:09.858531 18733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:09.871984 18733 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:40205
I20260620 19:09:09.872002 18861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:40205 every 8 connection(s)
I20260620 19:09:09.872958 18733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:09:09.873577 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 18733
I20260620 19:09:09.873788 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:09:09.874209 1642 external_mini_cluster.cc:1428] Setting key f971b6b3dbd04ac598af7583767336cf
Jun 20 19:09:09 dist-test-slave-6wrf krb5kdc[18626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982549, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
I20260620 19:09:09.891268 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:09.875144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:45881 (local address 127.1.154.190:38669)
0620 19:09:09.875303 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:09.875307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:09.876342 (+ 1035us) server_negotiation.cc:407] Connection header received
0620 19:09:09.878264 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:09.878268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:09.878321 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:09.878405 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:09.879905 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:09.880477 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:09.881792 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:09.882037 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:09.882151 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:09.884916 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:09.884919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:09.884921 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:09.884944 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:09.887074 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:09.888285 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:09.888290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:09.888292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:09.888552 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:09.889153 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:09.889160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:09.889165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:09.889553 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:09.889718 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:09.890932 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0620 19:09:09.891084 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 19:09:09.892151 18864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:09.875559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:38669 (local address 127.1.154.129:45881)
0620 19:09:09.876162 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:09.876226 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:09.878046 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:09.878569 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:09.878577 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:09.879222 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:09.879735 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:09.879749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:09.880630 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:09.880638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:09.881656 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:09.881671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:09.882192 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:09.882196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:09.882375 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:09.883028 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:09.883044 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:09.884709 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:09.887271 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:09.887280 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:09.887299 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:09.888067 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:09.888765 (+ 698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:09.888768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:09.888770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:09.889013 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:09.889794 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:09.889800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:09.890347 (+ 547us) client_negotiation.cc:769] Sending connection context
0620 19:09:09.890870 (+ 523us) client_negotiation.cc:240] Negotiation successful
0620 19:09:09.891277 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":182,"threads_started":1}
I20260620 19:09:09.893486 18862 heartbeater.cc:347] Connected to a master server at 127.1.154.190:38669
I20260620 19:09:09.893761 18862 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:09.894284 18862 heartbeater.cc:511] Master 127.1.154.190:38669 requested a full tablet report, sending...
I20260620 19:09:09.896000 18671 ts_manager.cc:194] Registered new tserver with Master: e7aa0e82b9914128ad9bfcf9e249b2e6 (127.1.154.129:40205)
I20260620 19:09:09.897228 18671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:45881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:09:09.910444 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:38669
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:10.027266 18867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:10.027544 18867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:10.027591 18867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:10.031620 18867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:10.031692 18867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:10.031755 18867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:09:10.036899 18867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:10.038149 18867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:10.039266 18867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:10.049191 18872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:10.049289 18875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:10.049516 18867 server_base.cc:1034] running on GCE node
W20260620 19:09:10.049463 18873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:10.050235 18867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:10.050907 18867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:10.052116 18867 hybrid_clock.cc:648] HybridClock initialized: now 1781982550052077 us; error 63 us; skew 500 ppm
Jun 20 19:09:10 dist-test-slave-6wrf krb5kdc[18626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:10.055330 18867 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:09:10.056237 18867 webserver.cc:460] Webserver started at http://127.1.154.130:45419/ using document root <none> and password file <none>
I20260620 19:09:10.056836 18867 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:10.056943 18867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:10.057204 18867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:10.059460 18867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "57d08461f09f4725a8ae71bb01bc04e1"
format_stamp: "Formatted at 2026-06-20 19:09:10 on dist-test-slave-6wrf"
server_key: "69cd1bef039a5860c79d423a7f448d7a"
server_key_iv: "bb892569bd3748362a2d2f07dad738b5"
server_key_version: "encryptionkey@0"
I20260620 19:09:10.060072 18867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "57d08461f09f4725a8ae71bb01bc04e1"
format_stamp: "Formatted at 2026-06-20 19:09:10 on dist-test-slave-6wrf"
server_key: "69cd1bef039a5860c79d423a7f448d7a"
server_key_iv: "bb892569bd3748362a2d2f07dad738b5"
server_key_version: "encryptionkey@0"
I20260620 19:09:10.064317 18867 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:09:10.067178 18882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:10.068568 18867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:09:10.068722 18867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "57d08461f09f4725a8ae71bb01bc04e1"
format_stamp: "Formatted at 2026-06-20 19:09:10 on dist-test-slave-6wrf"
server_key: "69cd1bef039a5860c79d423a7f448d7a"
server_key_iv: "bb892569bd3748362a2d2f07dad738b5"
server_key_version: "encryptionkey@0"
I20260620 19:09:10.068846 18867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:10.087492 18867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:10.091723 18867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:10.092061 18867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:10.092926 18867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:10.094177 18867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:10.094271 18867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:10.094357 18867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:10.094426 18867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:10.109412 18867 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:37195
I20260620 19:09:10.109504 18995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:37195 every 8 connection(s)
I20260620 19:09:10.110587 18867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:09:10.119499 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 18867
I20260620 19:09:10.119688 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:09:10.120088 1642 external_mini_cluster.cc:1428] Setting key 43e731c529b0724aedb76810556ea750
Jun 20 19:09:10 dist-test-slave-6wrf krb5kdc[18626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:10.129877 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.113112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:57565 (local address 127.1.154.190:38669)
0620 19:09:10.113345 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:10.113350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:10.114834 (+ 1484us) server_negotiation.cc:407] Connection header received
0620 19:09:10.116730 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:10.116734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:10.116818 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:10.116942 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:10.118824 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.119380 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.121180 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.121450 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.121580 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.124720 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:10.124725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:10.124730 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:10.124767 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:10.126828 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:10.127852 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:10.127863 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:10.127866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:10.127969 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:10.128488 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:10.128492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:10.128494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:10.128820 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:10.128983 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:10.129542 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 19:09:10.129699 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":89}
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
I20260620 19:09:10.130828 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.113813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:38669 (local address 127.1.154.130:57565)
0620 19:09:10.114595 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:10.114657 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:10.116314 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:10.117272 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:10.117282 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:10.117906 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:10.118607 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:10.118627 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.119543 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.119548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:10.121039 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:10.121050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.121652 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.121657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:10.121848 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.122563 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:10.122587 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:10.124532 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:10.127009 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:10.127019 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:10.127035 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:10.127682 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:10.128194 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:10.128197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:10.128198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:10.128367 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:10.128984 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:10.128992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:10.129270 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 19:09:10.129479 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 19:09:10.129860 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":233,"threads_started":1}
I20260620 19:09:10.132517 18996 heartbeater.cc:347] Connected to a master server at 127.1.154.190:38669
I20260620 19:09:10.132818 18996 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:10.133450 18996 heartbeater.cc:511] Master 127.1.154.190:38669 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
I20260620 19:09:10.134663 18671 ts_manager.cc:194] Registered new tserver with Master: 57d08461f09f4725a8ae71bb01bc04e1 (127.1.154.130:37195)
I20260620 19:09:10.135560 18671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:57565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:09:10.153796 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:38669
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:10.268293 19001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:10.268587 19001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:10.268633 19001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:10.272883 19001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:10.272959 19001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:10.273021 19001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:09:10.278506 19001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:10.279570 19001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:10.280906 19001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:10.291165 19009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:10.291129 19006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:10.291188 19007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:10.291538 19001 server_base.cc:1034] running on GCE node
I20260620 19:09:10.291972 19001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:10.292568 19001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:10.293700 19001 hybrid_clock.cc:648] HybridClock initialized: now 1781982550293644 us; error 76 us; skew 500 ppm
Jun 20 19:09:10 dist-test-slave-6wrf krb5kdc[18626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:10.296691 19001 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:09:10.297724 19001 webserver.cc:460] Webserver started at http://127.1.154.131:45219/ using document root <none> and password file <none>
I20260620 19:09:10.298314 19001 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:10.298404 19001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:10.298614 19001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:10.300652 19001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "cf212af209a041d4b1535380338479a7"
format_stamp: "Formatted at 2026-06-20 19:09:10 on dist-test-slave-6wrf"
server_key: "875de4d36b46d7b02186ffcf96096bdb"
server_key_iv: "a57896289663b73e698ca45364ca0073"
server_key_version: "encryptionkey@0"
I20260620 19:09:10.301224 19001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "cf212af209a041d4b1535380338479a7"
format_stamp: "Formatted at 2026-06-20 19:09:10 on dist-test-slave-6wrf"
server_key: "875de4d36b46d7b02186ffcf96096bdb"
server_key_iv: "a57896289663b73e698ca45364ca0073"
server_key_version: "encryptionkey@0"
I20260620 19:09:10.305084 19001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 19:09:10.307690 19016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:10.308935 19001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:09:10.309078 19001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "cf212af209a041d4b1535380338479a7"
format_stamp: "Formatted at 2026-06-20 19:09:10 on dist-test-slave-6wrf"
server_key: "875de4d36b46d7b02186ffcf96096bdb"
server_key_iv: "a57896289663b73e698ca45364ca0073"
server_key_version: "encryptionkey@0"
I20260620 19:09:10.309196 19001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:10.334988 19001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:10.338415 19001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:10.338624 19001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:10.339277 19001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:10.340116 19001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:10.340185 19001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:10.340261 19001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:10.340304 19001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:10.350577 19001 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:33547
I20260620 19:09:10.350593 19129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:33547 every 8 connection(s)
I20260620 19:09:10.351548 19001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:09:10.352665 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 19001
I20260620 19:09:10.352803 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:09:10.353178 1642 external_mini_cluster.cc:1428] Setting key ad77cef9416cfd9a0bacd5e5bc2341f1
Jun 20 19:09:10 dist-test-slave-6wrf krb5kdc[18626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:10.368851 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.353763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:58555 (local address 127.1.154.190:38669)
0620 19:09:10.353917 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:10.353921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:10.354891 (+ 970us) server_negotiation.cc:407] Connection header received
0620 19:09:10.356345 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:10.356350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:10.356431 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:10.356607 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:10.358130 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.358684 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.359903 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.360172 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.360308 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.364081 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:10.364084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:10.364087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:10.364146 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:10.365796 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:10.366610 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:10.366614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:10.366616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:10.366721 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:10.367389 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:10.367392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:10.367394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:10.367661 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:10.367770 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:10.368516 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 19:09:10.368691 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:09:10.369982 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.354228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:38669 (local address 127.1.154.131:58555)
0620 19:09:10.354733 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:10.354787 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:10.356056 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:10.356890 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:10.356901 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:10.357373 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:10.357962 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:10.357980 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.358820 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.358822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:10.359719 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:10.359727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.360457 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.360462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:10.360684 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.361438 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:10.361457 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:10.363815 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:10.365952 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:10.365961 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:10.365985 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:10.366462 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:10.366935 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:10.366939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:10.366941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:10.367244 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:10.367835 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:10.367846 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:10.368146 (+ 300us) client_negotiation.cc:769] Sending connection context
0620 19:09:10.368321 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 19:09:10.368626 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":149,"threads_started":1}
I20260620 19:09:10.371506 19130 heartbeater.cc:347] Connected to a master server at 127.1.154.190:38669
I20260620 19:09:10.371793 19130 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:10.372401 19130 heartbeater.cc:511] Master 127.1.154.190:38669 requested a full tablet report, sending...
I20260620 19:09:10.373438 18671 ts_manager.cc:194] Registered new tserver with Master: cf212af209a041d4b1535380338479a7 (127.1.154.131:33547)
I20260620 19:09:10.374064 18671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:58555
I20260620 19:09:10.380264 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:09:10.392109 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.382907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.1.154.190:38669)
0620 19:09:10.383150 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:10.383155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:10.383307 (+ 152us) server_negotiation.cc:407] Connection header received
0620 19:09:10.383491 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:10.383493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:10.383536 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:10.383619 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:10.384331 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.384852 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.385774 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.385993 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.386101 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.387960 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:10.387963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:10.387966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:10.387988 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:10.389954 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:10.390532 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:10.390535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:10.390536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:10.390632 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:10.390974 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:10.390978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:10.390979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:10.391304 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:10.391450 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:10.391725 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 19:09:10.391904 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260620 19:09:10.396433 18671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:09:10.399909 18671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:09:10.415724 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.410072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.1.154.129:40205)
0620 19:09:10.410470 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:10.410475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:10.410492 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:09:10.410553 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:10.410557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:10.410730 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:10.410897 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:10.411110 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.412947 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.414411 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.415062 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.415188 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.415335 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:10.415491 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 19:09:10.415579 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":120,"threads_started":1}
I20260620 19:09:10.416085 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.409588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:40205 (local address 127.0.0.1:38314)
0620 19:09:10.410135 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:10.410146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:10.410214 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:10.410868 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:10.410871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:10.410886 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:10.410957 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:10.410961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.413074 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.413080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:10.414282 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:10.414292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.415187 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.415190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:10.415293 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.415329 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 19:09:10.415446 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 19:09:10.415564 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":197,"thread_start_us":114,"threads_started":1}
I20260620 19:09:10.416461 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.409658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33304 (local address 127.1.154.131:33547)
0620 19:09:10.411169 (+ 1511us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:10.411178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:10.411195 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:09:10.411256 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:10.411261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:10.411383 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:10.411535 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:10.411774 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.412776 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.414622 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.415379 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.415511 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.415644 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:10.416151 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 19:09:10.416272 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1418,"thread_start_us":98,"threads_started":1}
I20260620 19:09:10.416738 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.409041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.1.154.130:37195)
0620 19:09:10.409563 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:10.409578 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:10.410116 (+ 538us) server_negotiation.cc:407] Connection header received
0620 19:09:10.410421 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:10.410426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:10.410581 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:10.410700 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:10.411054 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.412328 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.414487 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.415354 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.415511 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.415644 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:10.416498 (+ 854us) server_negotiation.cc:299] Negotiation successful
0620 19:09:10.416593 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":150,"threads_started":1}
I20260620 19:09:10.416735 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.409070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:33547 (local address 127.0.0.1:33304)
0620 19:09:10.410027 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:10.410043 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:10.410172 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:10.411517 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:10.411520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:10.411539 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:10.411631 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:10.411640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.412942 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.412946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:10.414508 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:10.414516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.415715 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.415719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:10.415902 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.415927 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 19:09:10.416003 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:09:10.416071 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":245,"thread_start_us":136,"threads_started":1}
I20260620 19:09:10.417112 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.408869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:37195 (local address 127.0.0.1:44802)
0620 19:09:10.409970 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:10.410022 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:10.410201 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:10.410732 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:10.410738 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:10.410761 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:10.410922 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:10.410930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.412619 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.412623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:10.414343 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:10.414353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.415715 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.415719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:10.416124 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.416313 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 19:09:10.416436 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 19:09:10.416673 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":118,"thread_start_us":81,"threads_started":1}
I20260620 19:09:10.419374 19064 tablet_service.cc:1467] Processing CreateTablet for tablet 7e86f60f1abe450aa2c0221445a6ec47 (DEFAULT_TABLE table=test-table [id=edb31a26630547c181057d7eb2308359]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:10.419517 18930 tablet_service.cc:1467] Processing CreateTablet for tablet 7e86f60f1abe450aa2c0221445a6ec47 (DEFAULT_TABLE table=test-table [id=edb31a26630547c181057d7eb2308359]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:10.419495 18796 tablet_service.cc:1467] Processing CreateTablet for tablet 7e86f60f1abe450aa2c0221445a6ec47 (DEFAULT_TABLE table=test-table [id=edb31a26630547c181057d7eb2308359]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:10.420311 19064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e86f60f1abe450aa2c0221445a6ec47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:10.420472 18796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e86f60f1abe450aa2c0221445a6ec47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:10.420472 18930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e86f60f1abe450aa2c0221445a6ec47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:10.427892 19147 tablet_bootstrap.cc:492] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7: Bootstrap starting.
I20260620 19:09:10.428303 19149 tablet_bootstrap.cc:492] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1: Bootstrap starting.
I20260620 19:09:10.429930 19148 tablet_bootstrap.cc:492] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6: Bootstrap starting.
I20260620 19:09:10.429960 19147 tablet_bootstrap.cc:654] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:10.430495 19149 tablet_bootstrap.cc:654] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:10.431059 19147 log.cc:824] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:10.431288 19149 log.cc:824] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:10.432068 19148 tablet_bootstrap.cc:654] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:10.432801 19148 log.cc:824] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:10.433256 19149 tablet_bootstrap.cc:492] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1: No bootstrap required, opened a new log
I20260620 19:09:10.433269 19147 tablet_bootstrap.cc:492] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7: No bootstrap required, opened a new log
I20260620 19:09:10.433454 19147 ts_tablet_manager.cc:1397] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:09:10.433454 19149 ts_tablet_manager.cc:1397] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:09:10.434976 19148 tablet_bootstrap.cc:492] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6: No bootstrap required, opened a new log
I20260620 19:09:10.435155 19148 ts_tablet_manager.cc:1397] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:09:10.436496 19149 raft_consensus.cc:348] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d08461f09f4725a8ae71bb01bc04e1" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37195 } } peers { permanent_uuid: "cf212af209a041d4b1535380338479a7" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33547 } } peers { permanent_uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40205 } }
I20260620 19:09:10.436744 19149 raft_consensus.cc:374] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:10.436795 19149 raft_consensus.cc:729] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57d08461f09f4725a8ae71bb01bc04e1, State: Initialized, Role: FOLLOWER
I20260620 19:09:10.437235 19149 consensus_queue.cc:260] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d08461f09f4725a8ae71bb01bc04e1" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37195 } } peers { permanent_uuid: "cf212af209a041d4b1535380338479a7" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33547 } } peers { permanent_uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40205 } }
I20260620 19:09:10.437305 19147 raft_consensus.cc:348] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d08461f09f4725a8ae71bb01bc04e1" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37195 } } peers { permanent_uuid: "cf212af209a041d4b1535380338479a7" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33547 } } peers { permanent_uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40205 } }
I20260620 19:09:10.437567 19147 raft_consensus.cc:374] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:10.437645 19147 raft_consensus.cc:729] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf212af209a041d4b1535380338479a7, State: Initialized, Role: FOLLOWER
I20260620 19:09:10.438026 18996 heartbeater.cc:503] Master 127.1.154.190:38669 was elected leader, sending a full tablet report...
I20260620 19:09:10.438056 19147 consensus_queue.cc:260] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d08461f09f4725a8ae71bb01bc04e1" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37195 } } peers { permanent_uuid: "cf212af209a041d4b1535380338479a7" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33547 } } peers { permanent_uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40205 } }
I20260620 19:09:10.438313 19149 ts_tablet_manager.cc:1428] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:09:10.438912 19147 ts_tablet_manager.cc:1428] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:09:10.439210 19130 heartbeater.cc:503] Master 127.1.154.190:38669 was elected leader, sending a full tablet report...
I20260620 19:09:10.439668 19148 raft_consensus.cc:348] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d08461f09f4725a8ae71bb01bc04e1" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37195 } } peers { permanent_uuid: "cf212af209a041d4b1535380338479a7" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33547 } } peers { permanent_uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40205 } }
I20260620 19:09:10.439905 19148 raft_consensus.cc:374] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:10.439957 19148 raft_consensus.cc:729] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7aa0e82b9914128ad9bfcf9e249b2e6, State: Initialized, Role: FOLLOWER
I20260620 19:09:10.440348 19148 consensus_queue.cc:260] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d08461f09f4725a8ae71bb01bc04e1" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37195 } } peers { permanent_uuid: "cf212af209a041d4b1535380338479a7" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33547 } } peers { permanent_uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40205 } }
I20260620 19:09:10.440996 18862 heartbeater.cc:503] Master 127.1.154.190:38669 was elected leader, sending a full tablet report...
I20260620 19:09:10.441218 19148 ts_tablet_manager.cc:1428] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:09:10.444501 19155 raft_consensus.cc:482] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:09:10.444621 19155 raft_consensus.cc:504] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d08461f09f4725a8ae71bb01bc04e1" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37195 } } peers { permanent_uuid: "cf212af209a041d4b1535380338479a7" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33547 } } peers { permanent_uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40205 } }
I20260620 19:09:10.445508 19155 leader_election.cc:288] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57d08461f09f4725a8ae71bb01bc04e1 (127.1.154.130:37195), cf212af209a041d4b1535380338479a7 (127.1.154.131:33547)
I20260620 19:09:10.450047 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.445817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:52297 (local address 127.1.154.130:37195)
0620 19:09:10.446100 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:10.446103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:10.446478 (+ 375us) server_negotiation.cc:407] Connection header received
0620 19:09:10.446640 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:10.446643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:10.446692 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:10.446769 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:10.447046 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.447547 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.448927 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.449448 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.449547 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.449585 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:10.449843 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 19:09:10.449915 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260620 19:09:10.450079 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.445779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:37195 (local address 127.1.154.129:52297)
0620 19:09:10.446389 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:10.446401 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:10.446529 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:10.446794 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:10.446797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:10.446813 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:10.446947 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:10.446952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.447653 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.447658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:10.448712 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:10.448722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.449570 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.449575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:10.449718 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.449745 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:09:10.449819 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:09:10.449895 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":332,"thread_start_us":104,"threads_started":1}
I20260620 19:09:10.450181 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.445889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:51311 (local address 127.1.154.131:33547)
0620 19:09:10.446278 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:10.446286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:10.446421 (+ 135us) server_negotiation.cc:407] Connection header received
0620 19:09:10.446835 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:10.446840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:10.446885 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:10.446972 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:10.447253 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.447743 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.449163 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.449635 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.449744 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.449800 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:10.449985 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 19:09:10.450051 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260620 19:09:10.450583 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.445779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:33547 (local address 127.1.154.129:51311)
0620 19:09:10.446278 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:10.446303 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:10.446511 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:10.447016 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:10.447019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:10.447037 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:10.447150 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:10.447155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.447854 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.447859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:10.449035 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:10.449045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.449739 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.449741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:10.449858 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.449881 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:09:10.449957 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:09:10.450029 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":136,"threads_started":1}
I20260620 19:09:10.450793 18950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e86f60f1abe450aa2c0221445a6ec47" candidate_uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d08461f09f4725a8ae71bb01bc04e1" is_pre_election: true
I20260620 19:09:10.451138 18950 raft_consensus.cc:2454] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7aa0e82b9914128ad9bfcf9e249b2e6 in term 0.
I20260620 19:09:10.451229 19084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e86f60f1abe450aa2c0221445a6ec47" candidate_uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf212af209a041d4b1535380338479a7" is_pre_election: true
I20260620 19:09:10.451494 19084 raft_consensus.cc:2454] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7aa0e82b9914128ad9bfcf9e249b2e6 in term 0.
I20260620 19:09:10.451963 18752 leader_election.cc:302] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 57d08461f09f4725a8ae71bb01bc04e1, cf212af209a041d4b1535380338479a7, e7aa0e82b9914128ad9bfcf9e249b2e6; no voters:
I20260620 19:09:10.452237 19155 raft_consensus.cc:2781] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:09:10.452311 19155 raft_consensus.cc:482] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:09:10.452342 19155 raft_consensus.cc:3037] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:10.453918 19155 raft_consensus.cc:504] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d08461f09f4725a8ae71bb01bc04e1" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37195 } } peers { permanent_uuid: "cf212af209a041d4b1535380338479a7" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33547 } } peers { permanent_uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40205 } }
I20260620 19:09:10.454334 19155 leader_election.cc:288] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [CANDIDATE]: Term 1 election: Requested vote from peers 57d08461f09f4725a8ae71bb01bc04e1 (127.1.154.130:37195), cf212af209a041d4b1535380338479a7 (127.1.154.131:33547)
I20260620 19:09:10.454766 18950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e86f60f1abe450aa2c0221445a6ec47" candidate_uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d08461f09f4725a8ae71bb01bc04e1"
I20260620 19:09:10.454766 19084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e86f60f1abe450aa2c0221445a6ec47" candidate_uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf212af209a041d4b1535380338479a7"
I20260620 19:09:10.454896 18950 raft_consensus.cc:3037] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:10.454896 19084 raft_consensus.cc:3037] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:10.456184 19084 raft_consensus.cc:2454] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7aa0e82b9914128ad9bfcf9e249b2e6 in term 1.
I20260620 19:09:10.456184 18950 raft_consensus.cc:2454] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7aa0e82b9914128ad9bfcf9e249b2e6 in term 1.
I20260620 19:09:10.456697 18751 leader_election.cc:302] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57d08461f09f4725a8ae71bb01bc04e1, e7aa0e82b9914128ad9bfcf9e249b2e6; no voters:
I20260620 19:09:10.457099 19155 raft_consensus.cc:2781] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:10.457495 19155 raft_consensus.cc:686] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [term 1 LEADER]: Becoming Leader. State: Replica: e7aa0e82b9914128ad9bfcf9e249b2e6, State: Running, Role: LEADER
I20260620 19:09:10.457999 19155 consensus_queue.cc:237] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d08461f09f4725a8ae71bb01bc04e1" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37195 } } peers { permanent_uuid: "cf212af209a041d4b1535380338479a7" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33547 } } peers { permanent_uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40205 } }
I20260620 19:09:10.462420 18671 catalog_manager.cc:5495] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7aa0e82b9914128ad9bfcf9e249b2e6 (127.1.154.129). New cstate: current_term: 1 leader_uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d08461f09f4725a8ae71bb01bc04e1" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf212af209a041d4b1535380338479a7" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7aa0e82b9914128ad9bfcf9e249b2e6" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 40205 } health_report { overall_health: HEALTHY } } }
I20260620 19:09:10.483773 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.479589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38324 (local address 127.1.154.129:40205)
0620 19:09:10.479773 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:10.479777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:10.479791 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:09:10.479876 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:10.479878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:10.479930 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:10.480028 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:10.480404 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.481151 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.482251 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.482509 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.482598 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.482871 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:10.483304 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:10.483393 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:10.483515 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 19:09:10.483567 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 19:09:10.489807 19084 raft_consensus.cc:1261] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7 [term 1 FOLLOWER]: Refusing update from remote peer e7aa0e82b9914128ad9bfcf9e249b2e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:10.490212 18950 raft_consensus.cc:1261] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1 [term 1 FOLLOWER]: Refusing update from remote peer e7aa0e82b9914128ad9bfcf9e249b2e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:10.490602 19158 consensus_queue.cc:1035] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf212af209a041d4b1535380338479a7" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 33547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:10.491032 19155 consensus_queue.cc:1035] T 7e86f60f1abe450aa2c0221445a6ec47 P e7aa0e82b9914128ad9bfcf9e249b2e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57d08461f09f4725a8ae71bb01bc04e1" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:10.499333 19162 mvcc.cc:205] Tried to move back new op lower bound from 7299000526798721024 to 7299000526680780800. Current Snapshot: MvccSnapshot[applied={T|T < 7299000526798721024}]
I20260620 19:09:10.499747 19163 mvcc.cc:205] Tried to move back new op lower bound from 7299000526798721024 to 7299000526680780800. Current Snapshot: MvccSnapshot[applied={T|T < 7299000526798721024}]
I20260620 19:09:10.507725 18667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:09:10.507879 18667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:09:10.510733 18667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d222434bb6684dd79391cc31975f8575: Sending DeleteTablet for 3 replicas of tablet 7e86f60f1abe450aa2c0221445a6ec47
I20260620 19:09:10.511705 19064 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e86f60f1abe450aa2c0221445a6ec47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:10 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:33304
I20260620 19:09:10.511929 18930 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e86f60f1abe450aa2c0221445a6ec47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:10 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:44802
I20260620 19:09:10.512378 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 18733
I20260620 19:09:10.512400 18796 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e86f60f1abe450aa2c0221445a6ec47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:10 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:38314
I20260620 19:09:10.513824 19171 tablet_replica.cc:330] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7: stopping tablet replica
I20260620 19:09:10.514122 19171 raft_consensus.cc:2229] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:09:10.514446 19171 raft_consensus.cc:2258] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:10.514519 19172 tablet_replica.cc:330] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1: stopping tablet replica
I20260620 19:09:10.514827 19172 raft_consensus.cc:2229] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:09:10.515098 19172 raft_consensus.cc:2258] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:10.515949 19171 ts_tablet_manager.cc:1905] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:09:10.516518 19172 ts_tablet_manager.cc:1905] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:09:10.518193 19171 ts_tablet_manager.cc:1918] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:09:10.518301 19171 log.cc:1196] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/7e86f60f1abe450aa2c0221445a6ec47
I20260620 19:09:10.518592 19171 ts_tablet_manager.cc:1939] T 7e86f60f1abe450aa2c0221445a6ec47 P cf212af209a041d4b1535380338479a7: Deleting consensus metadata
I20260620 19:09:10.519680 18658 catalog_manager.cc:4841] TS cf212af209a041d4b1535380338479a7 (127.1.154.131:33547): tablet 7e86f60f1abe450aa2c0221445a6ec47 (table test-table [id=edb31a26630547c181057d7eb2308359]) successfully deleted
I20260620 19:09:10.519908 19172 ts_tablet_manager.cc:1918] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:09:10.520020 19172 log.cc:1196] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/7e86f60f1abe450aa2c0221445a6ec47
I20260620 19:09:10.520358 19172 ts_tablet_manager.cc:1939] T 7e86f60f1abe450aa2c0221445a6ec47 P 57d08461f09f4725a8ae71bb01bc04e1: Deleting consensus metadata
I20260620 19:09:10.520498 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 18867
W20260620 19:09:10.521754 18657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:40205 (error 108)
I20260620 19:09:10.522951 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.522680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:40205 (local address 127.0.0.1:38328)
0620 19:09:10.522810 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:10.522880 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 19:09:10.523216 18657 catalog_manager.cc:4584] TS e7aa0e82b9914128ad9bfcf9e249b2e6 (127.1.154.129:40205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e86f60f1abe450aa2c0221445a6ec47: Network error: Client connection negotiation failed: client connection to 127.1.154.129:40205: connect: Connection refused (error 111)
I20260620 19:09:10.526050 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 19001
I20260620 19:09:10.527794 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.526194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:37195 (local address 127.0.0.1:44804)
0620 19:09:10.527680 (+ 1486us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:10.527729 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.130:37195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1414}
I20260620 19:09:10.531965 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 18639
2026-06-20T19:09:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2366 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T19:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:10Z Disabled control of system clock
I20260620 19:09:10.563409 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:36091
--webserver_interface=127.1.154.190
--webserver_port=0
--builtin_ntp_servers=127.1.154.148:44379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:36091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:09:10.679718 19177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:10.680052 19177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:10.680094 19177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:10.684146 19177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:10.684235 19177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:10.684254 19177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:10.684274 19177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:10.684293 19177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:10.689303 19177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:44379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:36091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:36091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:10.690492 19177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:10.691851 19177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:10.701435 19182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:10.701514 19185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:10.701582 19177 server_base.cc:1034] running on GCE node
W20260620 19:09:10.701432 19183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:10.702337 19177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:10.703460 19177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:10.704666 19177 hybrid_clock.cc:648] HybridClock initialized: now 1781982550704592 us; error 88 us; skew 500 ppm
I20260620 19:09:10.706954 19177 webserver.cc:460] Webserver started at http://127.1.154.190:37379/ using document root <none> and password file <none>
I20260620 19:09:10.707536 19177 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:10.707628 19177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:10.707835 19177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:10.709935 19177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "9851865112544c42b56b9e5c1f1409be"
format_stamp: "Formatted at 2026-06-20 19:09:10 on dist-test-slave-6wrf"
server_key: "2dc65562db0f869cd71680cf11cfde72"
server_key_iv: "a1b7b68d4698abc3598b8c8fa31c65a3"
server_key_version: "encryptionkey@0"
I20260620 19:09:10.710500 19177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "9851865112544c42b56b9e5c1f1409be"
format_stamp: "Formatted at 2026-06-20 19:09:10 on dist-test-slave-6wrf"
server_key: "2dc65562db0f869cd71680cf11cfde72"
server_key_iv: "a1b7b68d4698abc3598b8c8fa31c65a3"
server_key_version: "encryptionkey@0"
I20260620 19:09:10.714156 19177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 19:09:10.717118 19191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:10.718636 19177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:10.718792 19177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "9851865112544c42b56b9e5c1f1409be"
format_stamp: "Formatted at 2026-06-20 19:09:10 on dist-test-slave-6wrf"
server_key: "2dc65562db0f869cd71680cf11cfde72"
server_key_iv: "a1b7b68d4698abc3598b8c8fa31c65a3"
server_key_version: "encryptionkey@0"
I20260620 19:09:10.718910 19177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:10.730422 19177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:10.731109 19177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:10.731305 19177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:10.738803 19177 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:36091
I20260620 19:09:10.738824 19243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:36091 every 8 connection(s)
I20260620 19:09:10.739750 19177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:10.742906 19244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:10.748133 19244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be: Bootstrap starting.
I20260620 19:09:10.749892 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 19177
I20260620 19:09:10.750046 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:10.750303 1642 external_mini_cluster.cc:1428] Setting key 07ec7f48f125acb6fd3caae53be5f458
I20260620 19:09:10.750648 19244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:10.751577 19244 log.cc:824] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:10.753474 19244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be: No bootstrap required, opened a new log
I20260620 19:09:10.757381 19244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9851865112544c42b56b9e5c1f1409be" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36091 } }
I20260620 19:09:10.757645 19244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:10.757705 19244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9851865112544c42b56b9e5c1f1409be, State: Initialized, Role: FOLLOWER
I20260620 19:09:10.758164 19244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9851865112544c42b56b9e5c1f1409be" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36091 } }
I20260620 19:09:10.758323 19244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:10.758383 19244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:10.758500 19244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:10.759483 19244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9851865112544c42b56b9e5c1f1409be" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36091 } }
I20260620 19:09:10.759877 19244 leader_election.cc:302] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9851865112544c42b56b9e5c1f1409be; no voters:
I20260620 19:09:10.760236 19244 leader_election.cc:288] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:10.760313 19249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:10.760540 19249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [term 1 LEADER]: Becoming Leader. State: Replica: 9851865112544c42b56b9e5c1f1409be, State: Running, Role: LEADER
I20260620 19:09:10.760834 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.751465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44704 (local address 127.1.154.190:36091)
0620 19:09:10.752195 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:10.752215 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:10.752257 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:09:10.752935 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:10.752952 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:10.753275 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:10.754791 (+ 1516us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:10.755138 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.756043 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.756942 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.757528 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.757812 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:10.758872 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:10.758878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:10.758894 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:10.759287 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:10.759340 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:10.759360 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:10.759472 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:10.759624 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 19:09:10.759886 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":119,"threads_started":1}
I20260620 19:09:10.760836 19249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9851865112544c42b56b9e5c1f1409be" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36091 } }
I20260620 19:09:10.761514 19244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:10.762072 19249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9851865112544c42b56b9e5c1f1409be. Latest consensus state: current_term: 1 leader_uuid: "9851865112544c42b56b9e5c1f1409be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9851865112544c42b56b9e5c1f1409be" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36091 } } }
I20260620 19:09:10.762187 19249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:10.762820 19250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9851865112544c42b56b9e5c1f1409be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9851865112544c42b56b9e5c1f1409be" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36091 } } }
I20260620 19:09:10.762919 19250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:10.763350 19254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:10.766480 19254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:10.772948 19254 catalog_manager.cc:1347] Generated new cluster ID: 64041ba9d51448b5b59e29f851b184d2
I20260620 19:09:10.773064 19254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:10.792498 19254 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:10.793807 19254 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:10.804565 19254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be: Generated new TSK 0
I20260620 19:09:10.805397 19254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:10.810935 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:36091
--builtin_ntp_servers=127.1.154.148:44379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:09:10.925073 19268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:10.925369 19268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:10.925447 19268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:10.929541 19268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:10.929651 19268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:10.929750 19268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:09:10.934566 19268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:44379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:36091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:10.935670 19268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:10.937050 19268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:10.947726 19276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:10.947829 19274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:10.947724 19273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:10.948060 19268 server_base.cc:1034] running on GCE node
I20260620 19:09:10.948643 19268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:10.949303 19268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:10.950518 19268 hybrid_clock.cc:648] HybridClock initialized: now 1781982550950519 us; error 59 us; skew 500 ppm
I20260620 19:09:10.952642 19268 webserver.cc:460] Webserver started at http://127.1.154.129:35425/ using document root <none> and password file <none>
I20260620 19:09:10.953233 19268 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:10.953327 19268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:10.953558 19268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:10.956516 19268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "a72d73b915c248e4a1ffd3e03993d970"
format_stamp: "Formatted at 2026-06-20 19:09:10 on dist-test-slave-6wrf"
server_key: "5f8cf3dd59c0fb184cfc9bb38b90fcb3"
server_key_iv: "bb7499ca7e74c72c74c36aa898d3300c"
server_key_version: "encryptionkey@0"
I20260620 19:09:10.957185 19268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "a72d73b915c248e4a1ffd3e03993d970"
format_stamp: "Formatted at 2026-06-20 19:09:10 on dist-test-slave-6wrf"
server_key: "5f8cf3dd59c0fb184cfc9bb38b90fcb3"
server_key_iv: "bb7499ca7e74c72c74c36aa898d3300c"
server_key_version: "encryptionkey@0"
I20260620 19:09:10.961541 19268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 19:09:10.964605 19282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:10.966033 19268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:10.966271 19268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "a72d73b915c248e4a1ffd3e03993d970"
format_stamp: "Formatted at 2026-06-20 19:09:10 on dist-test-slave-6wrf"
server_key: "5f8cf3dd59c0fb184cfc9bb38b90fcb3"
server_key_iv: "bb7499ca7e74c72c74c36aa898d3300c"
server_key_version: "encryptionkey@0"
I20260620 19:09:10.966405 19268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:10.979624 19268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:10.980334 19268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:10.980527 19268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:10.981132 19268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:10.982090 19268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:10.982168 19268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:10.982249 19268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:10.982293 19268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:10.991679 19268 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:36737
I20260620 19:09:10.991706 19395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:36737 every 8 connection(s)
I20260620 19:09:10.992538 19268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:09:10.998373 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 19268
I20260620 19:09:10.998565 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:09:10.998884 1642 external_mini_cluster.cc:1428] Setting key 75a6d9f773ead13266d6b199a1bad699
I20260620 19:09:11.001024 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:36091
--builtin_ntp_servers=127.1.154.148:44379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:09:11.002553 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.994688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:33743 (local address 127.1.154.190:36091)
0620 19:09:10.994945 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:10.994950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:10.995741 (+ 791us) server_negotiation.cc:407] Connection header received
0620 19:09:10.997187 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:10.997190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:10.997257 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:10.997363 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:10.998268 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.998942 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.000333 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.000663 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.000790 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.001909 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:11.001916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:11.001922 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:11.001940 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:11.001966 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:11.001979 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:11.002061 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:11.002253 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 19:09:11.002330 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 19:09:11.003427 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:10.994821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:36091 (local address 127.1.154.129:33743)
0620 19:09:10.995527 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:10.995567 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:10.996965 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:10.997530 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:10.997536 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:10.997650 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:10.998118 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:10.998129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:10.999119 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:10.999122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.000155 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:11.000168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.000820 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.000823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.000959 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.001580 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:09:11.001582 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:11.001678 (+ 96us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:09:11.001758 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:11.002105 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:11.002112 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:11.002117 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:09:11.002223 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 19:09:11.002444 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":121,"threads_started":1}
I20260620 19:09:11.004689 19396 heartbeater.cc:347] Connected to a master server at 127.1.154.190:36091
I20260620 19:09:11.004944 19396 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:11.005563 19396 heartbeater.cc:511] Master 127.1.154.190:36091 requested a full tablet report, sending...
I20260620 19:09:11.007902 19208 ts_manager.cc:194] Registered new tserver with Master: a72d73b915c248e4a1ffd3e03993d970 (127.1.154.129:36737)
I20260620 19:09:11.009889 19208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.154.129:33743
W20260620 19:09:11.121361 19399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:11.121747 19399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:11.121827 19399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:11.126255 19399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:11.126358 19399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:11.126458 19399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:09:11.132380 19399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:44379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:36091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:11.134102 19399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:11.136193 19399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:11.147037 19405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:11.147105 19404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:11.147122 19407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:11.147383 19399 server_base.cc:1034] running on GCE node
I20260620 19:09:11.147787 19399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:11.148460 19399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:11.149694 19399 hybrid_clock.cc:648] HybridClock initialized: now 1781982551149670 us; error 36 us; skew 500 ppm
I20260620 19:09:11.151559 19399 webserver.cc:460] Webserver started at http://127.1.154.130:44011/ using document root <none> and password file <none>
I20260620 19:09:11.152150 19399 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:11.152240 19399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:11.152472 19399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:11.154685 19399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "15694509ab18412a85a90d7a5cd51d97"
format_stamp: "Formatted at 2026-06-20 19:09:11 on dist-test-slave-6wrf"
server_key: "fd47ed322d97839a4cb284a016caded2"
server_key_iv: "7ccd084a649e389784a7e2178eae8eb9"
server_key_version: "encryptionkey@0"
I20260620 19:09:11.155251 19399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "15694509ab18412a85a90d7a5cd51d97"
format_stamp: "Formatted at 2026-06-20 19:09:11 on dist-test-slave-6wrf"
server_key: "fd47ed322d97839a4cb284a016caded2"
server_key_iv: "7ccd084a649e389784a7e2178eae8eb9"
server_key_version: "encryptionkey@0"
I20260620 19:09:11.159128 19399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:09:11.161948 19413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:11.163414 19399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:09:11.163559 19399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "15694509ab18412a85a90d7a5cd51d97"
format_stamp: "Formatted at 2026-06-20 19:09:11 on dist-test-slave-6wrf"
server_key: "fd47ed322d97839a4cb284a016caded2"
server_key_iv: "7ccd084a649e389784a7e2178eae8eb9"
server_key_version: "encryptionkey@0"
I20260620 19:09:11.163734 19399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:11.177986 19399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:11.178683 19399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:11.178879 19399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:11.179526 19399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:11.180500 19399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:11.180573 19399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:11.180645 19399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:11.180697 19399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:11.192592 19399 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:39209
I20260620 19:09:11.192672 19526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:39209 every 8 connection(s)
I20260620 19:09:11.193799 19399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:09:11.200095 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 19399
I20260620 19:09:11.200362 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:09:11.200721 1642 external_mini_cluster.cc:1428] Setting key d76dc71807bda9b06698ae8a3ce0f4f8
I20260620 19:09:11.203347 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:36091
--builtin_ntp_servers=127.1.154.148:44379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:09:11.208114 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.198084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:55615 (local address 127.1.154.190:36091)
0620 19:09:11.198278 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:11.198283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:11.199855 (+ 1572us) server_negotiation.cc:407] Connection header received
0620 19:09:11.201500 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:11.201504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:11.201560 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:11.201701 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:11.203194 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.204048 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.205638 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.205875 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.206005 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.207390 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:11.207393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:11.207396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:11.207410 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:11.207431 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:11.207448 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:11.207533 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:11.207922 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 19:09:11.207984 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 19:09:11.209407 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.198105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:36091 (local address 127.1.154.130:55615)
0620 19:09:11.199562 (+ 1457us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:11.199610 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:11.201206 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:11.202043 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:11.202054 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:11.202252 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:11.203016 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:11.203034 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.204209 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.204215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.205472 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:11.205481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.206046 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.206050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.206280 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.207022 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:09:11.207027 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:11.207084 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:09:11.207200 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:11.207550 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:11.207561 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:11.207568 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 19:09:11.207720 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 19:09:11.208078 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":157,"threads_started":1}
I20260620 19:09:11.210702 19527 heartbeater.cc:347] Connected to a master server at 127.1.154.190:36091
I20260620 19:09:11.210978 19527 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:11.211676 19527 heartbeater.cc:511] Master 127.1.154.190:36091 requested a full tablet report, sending...
I20260620 19:09:11.212905 19208 ts_manager.cc:194] Registered new tserver with Master: 15694509ab18412a85a90d7a5cd51d97 (127.1.154.130:39209)
I20260620 19:09:11.213548 19208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.154.130:55615
W20260620 19:09:11.318964 19530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:11.319275 19530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:11.319344 19530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:11.323715 19530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:11.323803 19530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:11.323894 19530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:09:11.328841 19530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:44379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:36091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:11.330003 19530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:11.331403 19530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:11.344600 19536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:11.344518 19538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:11.344602 19535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:11.345016 19530 server_base.cc:1034] running on GCE node
I20260620 19:09:11.345405 19530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:11.346096 19530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:11.347311 19530 hybrid_clock.cc:648] HybridClock initialized: now 1781982551347289 us; error 31 us; skew 500 ppm
I20260620 19:09:11.349291 19530 webserver.cc:460] Webserver started at http://127.1.154.131:33471/ using document root <none> and password file <none>
I20260620 19:09:11.349947 19530 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:11.350013 19530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:11.350235 19530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:11.352303 19530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "e425f735de0c4beda155f22aa5c3f84b"
format_stamp: "Formatted at 2026-06-20 19:09:11 on dist-test-slave-6wrf"
server_key: "caa17e15ba43ce0840ba0258643f9088"
server_key_iv: "5691358b9bc0244ef1d23fe97f7afbd5"
server_key_version: "encryptionkey@0"
I20260620 19:09:11.352875 19530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "e425f735de0c4beda155f22aa5c3f84b"
format_stamp: "Formatted at 2026-06-20 19:09:11 on dist-test-slave-6wrf"
server_key: "caa17e15ba43ce0840ba0258643f9088"
server_key_iv: "5691358b9bc0244ef1d23fe97f7afbd5"
server_key_version: "encryptionkey@0"
I20260620 19:09:11.357565 19530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:09:11.360617 19544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:11.362751 19530 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:09:11.362957 19530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "e425f735de0c4beda155f22aa5c3f84b"
format_stamp: "Formatted at 2026-06-20 19:09:11 on dist-test-slave-6wrf"
server_key: "caa17e15ba43ce0840ba0258643f9088"
server_key_iv: "5691358b9bc0244ef1d23fe97f7afbd5"
server_key_version: "encryptionkey@0"
I20260620 19:09:11.363155 19530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:11.385437 19530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:11.386209 19530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:11.386415 19530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:11.387044 19530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:11.387977 19530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:11.388049 19530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:11.388134 19530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:11.388186 19530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:11.400627 19530 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:42091
I20260620 19:09:11.400653 19657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:42091 every 8 connection(s)
I20260620 19:09:11.401639 19530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:09:11.402248 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 19530
I20260620 19:09:11.402379 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:09:11.402727 1642 external_mini_cluster.cc:1428] Setting key e08b543f9069e4226a9028724e15baa2
I20260620 19:09:11.411404 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.403902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:59097 (local address 127.1.154.190:36091)
0620 19:09:11.404098 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:11.404103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:11.405485 (+ 1382us) server_negotiation.cc:407] Connection header received
0620 19:09:11.406892 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:11.406895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:11.406945 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:11.407023 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:11.407901 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.408441 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.409561 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.409803 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.409917 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.410921 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:11.410925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:11.410927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:11.410947 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:11.410966 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:11.410973 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:11.411053 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:11.411203 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 19:09:11.411264 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 19:09:11.412284 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.404361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:36091 (local address 127.1.154.131:59097)
0620 19:09:11.405328 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:11.405380 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:11.406695 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:11.407220 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:11.407226 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:11.407306 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:11.407760 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:11.407771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.408568 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.408572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.409428 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:11.409436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.409948 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.409951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.410112 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.410681 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:09:11.410683 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:11.410721 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:09:11.410778 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:11.411055 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:11.411065 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:11.411073 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:09:11.411220 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 19:09:11.411484 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":171,"threads_started":1}
I20260620 19:09:11.413831 19658 heartbeater.cc:347] Connected to a master server at 127.1.154.190:36091
I20260620 19:09:11.414098 19658 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:11.414712 19658 heartbeater.cc:511] Master 127.1.154.190:36091 requested a full tablet report, sending...
I20260620 19:09:11.416162 19208 ts_manager.cc:194] Registered new tserver with Master: e425f735de0c4beda155f22aa5c3f84b (127.1.154.131:42091)
I20260620 19:09:11.417176 19208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.154.131:59097
I20260620 19:09:11.418084 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:09:11.424487 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.420399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.1.154.190:36091)
0620 19:09:11.420540 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:11.420551 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:11.420565 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:09:11.420601 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:11.420603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:11.420641 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:11.420722 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:11.421266 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.421929 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.422859 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.423073 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.423153 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.424126 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:11.424129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:11.424131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:11.424142 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:11.424158 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:11.424165 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:11.424227 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:11.424324 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 19:09:11.424371 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 19:09:11.427592 19208 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:09:11.430653 19208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:09:11.446169 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.440713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.1.154.131:42091)
0620 19:09:11.441262 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:11.441273 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:11.441564 (+ 291us) server_negotiation.cc:407] Connection header received
0620 19:09:11.441671 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:11.441676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:11.441829 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:11.441983 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:11.442293 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.443092 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.444522 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.445118 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.445251 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.445434 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:11.445960 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 19:09:11.446039 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":132,"threads_started":1}
I20260620 19:09:11.446245 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.440537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:42091 (local address 127.0.0.1:36396)
0620 19:09:11.441350 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:11.441364 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:11.441467 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:11.441988 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:11.441993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:11.442014 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:11.442149 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:11.442155 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.443216 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.443223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.444402 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:11.444411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.445516 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.445519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.445660 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.445702 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:09:11.445817 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 19:09:11.445929 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":109,"threads_started":1}
I20260620 19:09:11.447628 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.440761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:36737 (local address 127.0.0.1:56574)
0620 19:09:11.441398 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:11.441412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:11.441476 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:11.442247 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:11.442251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:11.442274 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:11.442360 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:11.442366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.443747 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.443750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.445116 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:11.445127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.447165 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.447170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.447263 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.447290 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:09:11.447377 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 19:09:11.447449 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":98,"thread_start_us":89,"threads_started":1}
I20260620 19:09:11.448073 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.440062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:39209 (local address 127.0.0.1:57764)
0620 19:09:11.441239 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:11.441285 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:11.441410 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:11.442142 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:11.442145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:11.442167 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:11.442295 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:11.442302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.443747 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.443750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.445123 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:11.445128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.447663 (+ 2535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.447666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.447765 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.447787 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:09:11.447848 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:09:11.447909 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"mutex_wait_us":146,"thread_start_us":83,"threads_started":1}
I20260620 19:09:11.448163 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.440324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.1.154.130:39209)
0620 19:09:11.441052 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:11.441065 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:11.441550 (+ 485us) server_negotiation.cc:407] Connection header received
0620 19:09:11.441675 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:11.441682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:11.441867 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:11.442005 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:11.442429 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.443587 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.446681 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.447543 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.447661 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.447850 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:11.447898 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 19:09:11.447978 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":147,"threads_started":1}
I20260620 19:09:11.448360 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.440973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.1.154.129:36737)
0620 19:09:11.441826 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:11.441837 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:11.441859 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:09:11.441934 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:11.441941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:11.442108 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:11.442256 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:11.442454 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.443629 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.445245 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.446254 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.446903 (+ 649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.447093 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:11.447870 (+ 777us) server_negotiation.cc:299] Negotiation successful
0620 19:09:11.447973 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":139,"threads_started":1}
I20260620 19:09:11.449807 19592 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd7456151db4dfda0f51d3b9cc5a6b0 (DEFAULT_TABLE table=test-table [id=948b8416ba7947459da3ebf9e2b8444e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:11.450948 19592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd7456151db4dfda0f51d3b9cc5a6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:11.451310 19330 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd7456151db4dfda0f51d3b9cc5a6b0 (DEFAULT_TABLE table=test-table [id=948b8416ba7947459da3ebf9e2b8444e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:11.452201 19330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd7456151db4dfda0f51d3b9cc5a6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:11.451870 19461 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd7456151db4dfda0f51d3b9cc5a6b0 (DEFAULT_TABLE table=test-table [id=948b8416ba7947459da3ebf9e2b8444e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:11.453121 19461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd7456151db4dfda0f51d3b9cc5a6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:11.458894 19675 tablet_bootstrap.cc:492] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97: Bootstrap starting.
I20260620 19:09:11.460157 19676 tablet_bootstrap.cc:492] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b: Bootstrap starting.
I20260620 19:09:11.460357 19677 tablet_bootstrap.cc:492] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970: Bootstrap starting.
I20260620 19:09:11.461568 19675 tablet_bootstrap.cc:654] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:11.461946 19676 tablet_bootstrap.cc:654] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:11.462296 19677 tablet_bootstrap.cc:654] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:11.462463 19675 log.cc:824] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:11.462961 19676 log.cc:824] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:11.463071 19677 log.cc:824] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:11.465061 19675 tablet_bootstrap.cc:492] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97: No bootstrap required, opened a new log
I20260620 19:09:11.465167 19676 tablet_bootstrap.cc:492] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b: No bootstrap required, opened a new log
I20260620 19:09:11.465233 19675 ts_tablet_manager.cc:1397] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:09:11.465356 19676 ts_tablet_manager.cc:1397] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:09:11.467346 19677 tablet_bootstrap.cc:492] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970: No bootstrap required, opened a new log
I20260620 19:09:11.467658 19677 ts_tablet_manager.cc:1397] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 19:09:11.469309 19676 raft_consensus.cc:348] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15694509ab18412a85a90d7a5cd51d97" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39209 } } peers { permanent_uuid: "a72d73b915c248e4a1ffd3e03993d970" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 36737 } } peers { permanent_uuid: "e425f735de0c4beda155f22aa5c3f84b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42091 } }
I20260620 19:09:11.469691 19676 raft_consensus.cc:374] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:11.469751 19676 raft_consensus.cc:729] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e425f735de0c4beda155f22aa5c3f84b, State: Initialized, Role: FOLLOWER
I20260620 19:09:11.469625 19675 raft_consensus.cc:348] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15694509ab18412a85a90d7a5cd51d97" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39209 } } peers { permanent_uuid: "a72d73b915c248e4a1ffd3e03993d970" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 36737 } } peers { permanent_uuid: "e425f735de0c4beda155f22aa5c3f84b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42091 } }
I20260620 19:09:11.469868 19675 raft_consensus.cc:374] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:11.469918 19675 raft_consensus.cc:729] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15694509ab18412a85a90d7a5cd51d97, State: Initialized, Role: FOLLOWER
I20260620 19:09:11.470240 19676 consensus_queue.cc:260] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15694509ab18412a85a90d7a5cd51d97" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39209 } } peers { permanent_uuid: "a72d73b915c248e4a1ffd3e03993d970" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 36737 } } peers { permanent_uuid: "e425f735de0c4beda155f22aa5c3f84b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42091 } }
I20260620 19:09:11.470420 19675 consensus_queue.cc:260] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15694509ab18412a85a90d7a5cd51d97" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39209 } } peers { permanent_uuid: "a72d73b915c248e4a1ffd3e03993d970" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 36737 } } peers { permanent_uuid: "e425f735de0c4beda155f22aa5c3f84b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42091 } }
I20260620 19:09:11.471330 19676 ts_tablet_manager.cc:1428] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 19:09:11.471412 19675 ts_tablet_manager.cc:1428] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260620 19:09:11.471725 19658 heartbeater.cc:503] Master 127.1.154.190:36091 was elected leader, sending a full tablet report...
I20260620 19:09:11.471771 19527 heartbeater.cc:503] Master 127.1.154.190:36091 was elected leader, sending a full tablet report...
I20260620 19:09:11.472083 19677 raft_consensus.cc:348] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15694509ab18412a85a90d7a5cd51d97" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39209 } } peers { permanent_uuid: "a72d73b915c248e4a1ffd3e03993d970" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 36737 } } peers { permanent_uuid: "e425f735de0c4beda155f22aa5c3f84b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42091 } }
I20260620 19:09:11.472345 19677 raft_consensus.cc:374] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:11.472406 19677 raft_consensus.cc:729] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72d73b915c248e4a1ffd3e03993d970, State: Initialized, Role: FOLLOWER
I20260620 19:09:11.472822 19677 consensus_queue.cc:260] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15694509ab18412a85a90d7a5cd51d97" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39209 } } peers { permanent_uuid: "a72d73b915c248e4a1ffd3e03993d970" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 36737 } } peers { permanent_uuid: "e425f735de0c4beda155f22aa5c3f84b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42091 } }
I20260620 19:09:11.473567 19396 heartbeater.cc:503] Master 127.1.154.190:36091 was elected leader, sending a full tablet report...
I20260620 19:09:11.473840 19677 ts_tablet_manager.cc:1428] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 19:09:11.493777 19397 tablet.cc:2369] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:09:11.565471 19683 raft_consensus.cc:482] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:09:11.565827 19683 raft_consensus.cc:504] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15694509ab18412a85a90d7a5cd51d97" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39209 } } peers { permanent_uuid: "a72d73b915c248e4a1ffd3e03993d970" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 36737 } } peers { permanent_uuid: "e425f735de0c4beda155f22aa5c3f84b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42091 } }
I20260620 19:09:11.567029 19683 leader_election.cc:288] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15694509ab18412a85a90d7a5cd51d97 (127.1.154.130:39209), e425f735de0c4beda155f22aa5c3f84b (127.1.154.131:42091)
I20260620 19:09:11.571698 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.567369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:37909 (local address 127.1.154.130:39209)
0620 19:09:11.567539 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:11.567544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:11.567778 (+ 234us) server_negotiation.cc:407] Connection header received
0620 19:09:11.567952 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:11.567955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:11.568015 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:11.568081 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:11.568429 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.569043 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.570495 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.570997 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.571135 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.571189 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:11.571479 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 19:09:11.571573 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 19:09:11.571710 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.567252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:39209 (local address 127.1.154.129:37909)
0620 19:09:11.567665 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:11.567680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:11.567792 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:11.568131 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:11.568134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:11.568150 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:11.568317 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:11.568323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.569147 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.569154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.570346 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:11.570355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.571119 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.571123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.571216 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.571353 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 19:09:11.571435 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 19:09:11.571503 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":45,"thread_start_us":110,"threads_started":1}
I20260620 19:09:11.571825 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.567519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:57669 (local address 127.1.154.131:42091)
0620 19:09:11.567640 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:11.567644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:11.567844 (+ 200us) server_negotiation.cc:407] Connection header received
0620 19:09:11.567989 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:11.567991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:11.568030 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:11.568136 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:11.568463 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.569115 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.570463 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.571022 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.571155 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.571206 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:11.571628 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 19:09:11.571700 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:09:11.571866 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.567377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:42091 (local address 127.1.154.129:57669)
0620 19:09:11.567723 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:11.567754 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:11.567885 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:11.568149 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:11.568152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:11.568191 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:11.568327 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:11.568332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.569253 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.569256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.570342 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:11.570351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.571175 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.571179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:11.571452 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.571479 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:09:11.571577 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 19:09:11.571664 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":98,"spinlock_wait_cycles":22912,"thread_start_us":130,"threads_started":1}
I20260620 19:09:11.572561 19481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd7456151db4dfda0f51d3b9cc5a6b0" candidate_uuid: "a72d73b915c248e4a1ffd3e03993d970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15694509ab18412a85a90d7a5cd51d97" is_pre_election: true
I20260620 19:09:11.572860 19612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd7456151db4dfda0f51d3b9cc5a6b0" candidate_uuid: "a72d73b915c248e4a1ffd3e03993d970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e425f735de0c4beda155f22aa5c3f84b" is_pre_election: true
I20260620 19:09:11.572878 19481 raft_consensus.cc:2454] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72d73b915c248e4a1ffd3e03993d970 in term 0.
I20260620 19:09:11.573113 19612 raft_consensus.cc:2454] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72d73b915c248e4a1ffd3e03993d970 in term 0.
I20260620 19:09:11.573426 19285 leader_election.cc:302] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15694509ab18412a85a90d7a5cd51d97, a72d73b915c248e4a1ffd3e03993d970; no voters:
I20260620 19:09:11.573760 19683 raft_consensus.cc:2781] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:09:11.573833 19683 raft_consensus.cc:482] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:09:11.573865 19683 raft_consensus.cc:3037] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:11.574923 19683 raft_consensus.cc:504] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15694509ab18412a85a90d7a5cd51d97" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39209 } } peers { permanent_uuid: "a72d73b915c248e4a1ffd3e03993d970" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 36737 } } peers { permanent_uuid: "e425f735de0c4beda155f22aa5c3f84b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42091 } }
I20260620 19:09:11.575301 19683 leader_election.cc:288] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [CANDIDATE]: Term 1 election: Requested vote from peers 15694509ab18412a85a90d7a5cd51d97 (127.1.154.130:39209), e425f735de0c4beda155f22aa5c3f84b (127.1.154.131:42091)
I20260620 19:09:11.575668 19612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd7456151db4dfda0f51d3b9cc5a6b0" candidate_uuid: "a72d73b915c248e4a1ffd3e03993d970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e425f735de0c4beda155f22aa5c3f84b"
I20260620 19:09:11.575738 19481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd7456151db4dfda0f51d3b9cc5a6b0" candidate_uuid: "a72d73b915c248e4a1ffd3e03993d970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15694509ab18412a85a90d7a5cd51d97"
I20260620 19:09:11.575793 19612 raft_consensus.cc:3037] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:11.575842 19481 raft_consensus.cc:3037] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:11.577060 19612 raft_consensus.cc:2454] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72d73b915c248e4a1ffd3e03993d970 in term 1.
I20260620 19:09:11.577060 19481 raft_consensus.cc:2454] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72d73b915c248e4a1ffd3e03993d970 in term 1.
I20260620 19:09:11.577370 19283 leader_election.cc:302] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72d73b915c248e4a1ffd3e03993d970, e425f735de0c4beda155f22aa5c3f84b; no voters:
I20260620 19:09:11.577579 19683 raft_consensus.cc:2781] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:11.577834 19683 raft_consensus.cc:686] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [term 1 LEADER]: Becoming Leader. State: Replica: a72d73b915c248e4a1ffd3e03993d970, State: Running, Role: LEADER
I20260620 19:09:11.578195 19683 consensus_queue.cc:237] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15694509ab18412a85a90d7a5cd51d97" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39209 } } peers { permanent_uuid: "a72d73b915c248e4a1ffd3e03993d970" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 36737 } } peers { permanent_uuid: "e425f735de0c4beda155f22aa5c3f84b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42091 } }
I20260620 19:09:11.581790 19208 catalog_manager.cc:5495] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 reported cstate change: term changed from 0 to 1, leader changed from <none> to a72d73b915c248e4a1ffd3e03993d970 (127.1.154.129). New cstate: current_term: 1 leader_uuid: "a72d73b915c248e4a1ffd3e03993d970" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15694509ab18412a85a90d7a5cd51d97" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a72d73b915c248e4a1ffd3e03993d970" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e425f735de0c4beda155f22aa5c3f84b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42091 } health_report { overall_health: UNKNOWN } } }
I20260620 19:09:11.605142 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.600589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.1.154.129:36737)
0620 19:09:11.600824 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:11.600829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:11.600850 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:09:11.600970 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:11.600972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:11.601066 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:11.601195 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:11.601580 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.602255 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.603246 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:11.603521 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:11.603626 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:11.604099 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:11.604566 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:11.604666 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:11.604828 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 19:09:11.604961 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 19:09:11.611075 19612 raft_consensus.cc:1261] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b [term 1 FOLLOWER]: Refusing update from remote peer a72d73b915c248e4a1ffd3e03993d970: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:11.611075 19481 raft_consensus.cc:1261] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97 [term 1 FOLLOWER]: Refusing update from remote peer a72d73b915c248e4a1ffd3e03993d970: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:11.611872 19683 consensus_queue.cc:1035] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e425f735de0c4beda155f22aa5c3f84b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 42091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:11.612092 19686 consensus_queue.cc:1035] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15694509ab18412a85a90d7a5cd51d97" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:11.621605 19690 mvcc.cc:205] Tried to move back new op lower bound from 7299000531391254528 to 7299000531266932736. Current Snapshot: MvccSnapshot[applied={T|T < 7299000531391254528}]
I20260620 19:09:11.630764 19208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:09:11.630971 19208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:09:11.634594 19208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9851865112544c42b56b9e5c1f1409be: Sending DeleteTablet for 3 replicas of tablet 3fd7456151db4dfda0f51d3b9cc5a6b0
I20260620 19:09:11.635821 19592 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd7456151db4dfda0f51d3b9cc5a6b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:11 UTC) from {username='slave'} at 127.0.0.1:36396
I20260620 19:09:11.635839 19330 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd7456151db4dfda0f51d3b9cc5a6b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:11 UTC) from {username='slave'} at 127.0.0.1:56574
I20260620 19:09:11.635812 19461 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd7456151db4dfda0f51d3b9cc5a6b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:11 UTC) from {username='slave'} at 127.0.0.1:57764
I20260620 19:09:11.636427 19699 tablet_replica.cc:330] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b: stopping tablet replica
I20260620 19:09:11.636776 19700 tablet_replica.cc:330] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P a72d73b915c248e4a1ffd3e03993d970: stopping tablet replica
I20260620 19:09:11.636788 19699 raft_consensus.cc:2229] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:09:11.637120 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 19268
I20260620 19:09:11.638911 19701 tablet_replica.cc:330] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97: stopping tablet replica
I20260620 19:09:11.639158 19699 raft_consensus.cc:2258] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:11.639259 19701 raft_consensus.cc:2229] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:09:11.639568 19701 raft_consensus.cc:2258] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:11.640813 19699 ts_tablet_manager.cc:1905] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:09:11.640969 19701 ts_tablet_manager.cc:1905] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:09:11.643503 19699 ts_tablet_manager.cc:1918] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:09:11.643646 19699 log.cc:1196] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/3fd7456151db4dfda0f51d3b9cc5a6b0
I20260620 19:09:11.643786 19701 ts_tablet_manager.cc:1918] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:09:11.643884 19701 log.cc:1196] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/3fd7456151db4dfda0f51d3b9cc5a6b0
I20260620 19:09:11.644011 19699 ts_tablet_manager.cc:1939] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P e425f735de0c4beda155f22aa5c3f84b: Deleting consensus metadata
I20260620 19:09:11.644204 19701 ts_tablet_manager.cc:1939] T 3fd7456151db4dfda0f51d3b9cc5a6b0 P 15694509ab18412a85a90d7a5cd51d97: Deleting consensus metadata
W20260620 19:09:11.644958 19194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:36737 (error 108)
I20260620 19:09:11.645481 19192 catalog_manager.cc:4841] TS e425f735de0c4beda155f22aa5c3f84b (127.1.154.131:42091): tablet 3fd7456151db4dfda0f51d3b9cc5a6b0 (table test-table [id=948b8416ba7947459da3ebf9e2b8444e]) successfully deleted
I20260620 19:09:11.645517 19194 catalog_manager.cc:4841] TS 15694509ab18412a85a90d7a5cd51d97 (127.1.154.130:39209): tablet 3fd7456151db4dfda0f51d3b9cc5a6b0 (table test-table [id=948b8416ba7947459da3ebf9e2b8444e]) successfully deleted
I20260620 19:09:11.646047 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 19399
I20260620 19:09:11.646615 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:11.646251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:36737 (local address 127.0.0.1:56580)
0620 19:09:11.646446 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:11.646516 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260620 19:09:11.646811 19194 catalog_manager.cc:4584] TS a72d73b915c248e4a1ffd3e03993d970 (127.1.154.129:36737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fd7456151db4dfda0f51d3b9cc5a6b0: Network error: Client connection negotiation failed: client connection to 127.1.154.129:36737: connect: Connection refused (error 111)
I20260620 19:09:11.652077 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 19530
I20260620 19:09:11.658633 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 19177
2026-06-20T19:09:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1125 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:11 dist-test-slave-6wrf krb5kdc[19705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:11 dist-test-slave-6wrf krb5kdc[19705](info): set up 2 sockets
Jun 20 19:09:11 dist-test-slave-6wrf krb5kdc[19705](info): commencing operation
krb5kdc: starting...
W20260620 19:09:12.745671 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:12 dist-test-slave-6wrf krb5kdc[19705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:09:12.876868 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:34921
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:34921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:12.988080 19718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:12.988405 19718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:12.988481 19718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:12.992624 19718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:12.992727 19718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:12.992766 19718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:12.992805 19718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:12.992847 19718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:12.998988 19718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:34921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:34921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:13.000082 19718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:13.001132 19718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:13.009728 19723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:13.009743 19726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:13.009915 19724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:13.010237 19718 server_base.cc:1034] running on GCE node
I20260620 19:09:13.010809 19718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:13.011953 19718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:13.013159 19718 hybrid_clock.cc:648] HybridClock initialized: now 1781982553013136 us; error 45 us; skew 500 ppm
Jun 20 19:09:13 dist-test-slave-6wrf krb5kdc[19705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982553, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:13.016458 19718 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:09:13.017472 19718 webserver.cc:460] Webserver started at http://127.1.154.190:46115/ using document root <none> and password file <none>
I20260620 19:09:13.018102 19718 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:13.018195 19718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:13.018536 19718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:13.020653 19718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "760633c5ea4844ac88c9f3c447ec9eff"
format_stamp: "Formatted at 2026-06-20 19:09:13 on dist-test-slave-6wrf"
server_key: "36fd3d8ce95ee761736712b839a16e0f"
server_key_iv: "0eb6090cabffe57b933f551fccdeef38"
server_key_version: "encryptionkey@0"
I20260620 19:09:13.021219 19718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "760633c5ea4844ac88c9f3c447ec9eff"
format_stamp: "Formatted at 2026-06-20 19:09:13 on dist-test-slave-6wrf"
server_key: "36fd3d8ce95ee761736712b839a16e0f"
server_key_iv: "0eb6090cabffe57b933f551fccdeef38"
server_key_version: "encryptionkey@0"
I20260620 19:09:13.025282 19718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 19:09:13.028280 19733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:13.029755 19718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:13.029889 19718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "760633c5ea4844ac88c9f3c447ec9eff"
format_stamp: "Formatted at 2026-06-20 19:09:13 on dist-test-slave-6wrf"
server_key: "36fd3d8ce95ee761736712b839a16e0f"
server_key_iv: "0eb6090cabffe57b933f551fccdeef38"
server_key_version: "encryptionkey@0"
I20260620 19:09:13.030025 19718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:13.041224 19718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:13.044715 19718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:13.044932 19718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:13.054682 19718 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:34921
I20260620 19:09:13.054744 19785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:34921 every 8 connection(s)
I20260620 19:09:13.055698 19718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:13.058650 19786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:13.064108 19786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff: Bootstrap starting.
I20260620 19:09:13.064127 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 19718
I20260620 19:09:13.064301 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:13.064577 1642 external_mini_cluster.cc:1428] Setting key 1cd717a6c374cd4b594d3892138b4425
I20260620 19:09:13.066792 19786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:13.067657 19786 log.cc:824] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:13.070597 19786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff: No bootstrap required, opened a new log
Jun 20 19:09:13 dist-test-slave-6wrf krb5kdc[19705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:13.075350 19786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "760633c5ea4844ac88c9f3c447ec9eff" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 34921 } }
I20260620 19:09:13.075832 19786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:13.075917 19786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 760633c5ea4844ac88c9f3c447ec9eff, State: Initialized, Role: FOLLOWER
I20260620 19:09:13.076722 19786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "760633c5ea4844ac88c9f3c447ec9eff" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 34921 } }
I20260620 19:09:13.076933 19786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:13.077047 19786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:13.077265 19786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:13.078656 19786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "760633c5ea4844ac88c9f3c447ec9eff" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 34921 } }
I20260620 19:09:13.079156 19786 leader_election.cc:302] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 760633c5ea4844ac88c9f3c447ec9eff; no voters:
I20260620 19:09:13.079682 19786 leader_election.cc:288] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:13.079981 19791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:13.080276 19791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [term 1 LEADER]: Becoming Leader. State: Replica: 760633c5ea4844ac88c9f3c447ec9eff, State: Running, Role: LEADER
I20260620 19:09:13.080729 19791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "760633c5ea4844ac88c9f3c447ec9eff" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 34921 } }
I20260620 19:09:13.081637 19786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:13.082989 19793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 760633c5ea4844ac88c9f3c447ec9eff. Latest consensus state: current_term: 1 leader_uuid: "760633c5ea4844ac88c9f3c447ec9eff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "760633c5ea4844ac88c9f3c447ec9eff" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 34921 } } }
I20260620 19:09:13.083127 19793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:13.083810 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.066008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.1.154.190:34921)
0620 19:09:13.066431 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:13.066449 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:13.066488 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:09:13.067358 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:13.067381 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:13.067809 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:13.068170 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:13.069809 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.071271 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.072227 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.072927 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.073092 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.076745 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:13.076750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:13.076766 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:13.076797 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:13.079043 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:13.079943 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:13.079951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:13.079959 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:13.080396 (+ 437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:13.080809 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:13.080813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:13.080815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:13.081438 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:13.081730 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:13.082043 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 19:09:13.082467 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":285,"thread_start_us":114,"threads_started":1}
I20260620 19:09:13.084968 19792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "760633c5ea4844ac88c9f3c447ec9eff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "760633c5ea4844ac88c9f3c447ec9eff" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 34921 } } }
I20260620 19:09:13.085129 19792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff [sys.catalog]: This master's current role is: LEADER
W20260620 19:09:13.087450 19806 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:09:13.087553 19806 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:09:13.087636 19801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:13.088369 19801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:13.094369 19801 catalog_manager.cc:1347] Generated new cluster ID: 278639b563d24719957b84a257ab609b
I20260620 19:09:13.094483 19801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:13.107618 19801 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:13.108569 19801 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:13.116250 19801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff: Generated new TSK 0
I20260620 19:09:13.117079 19801 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:09:13.151379 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:34921
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:13.290601 19812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:13.290904 19812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:13.290982 19812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:13.296622 19812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:13.296726 19812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:13.296814 19812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:09:13.302850 19812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:34921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:13.303896 19812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:13.304844 19812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:13.315925 19817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:13.315954 19820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:13.315929 19818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:13.316316 19812 server_base.cc:1034] running on GCE node
I20260620 19:09:13.316782 19812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:13.317440 19812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:13.318670 19812 hybrid_clock.cc:648] HybridClock initialized: now 1781982553318647 us; error 50 us; skew 500 ppm
Jun 20 19:09:13 dist-test-slave-6wrf krb5kdc[19705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982553, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:13.326357 19812 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:09:13.327337 19812 webserver.cc:460] Webserver started at http://127.1.154.129:37463/ using document root <none> and password file <none>
I20260620 19:09:13.327873 19812 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:13.327936 19812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:13.328099 19812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:13.330368 19812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "b0a5d2093aa142b2a942f31a9c682fe1"
format_stamp: "Formatted at 2026-06-20 19:09:13 on dist-test-slave-6wrf"
server_key: "ebab4b148dd23855d0353cbfd2bc53cb"
server_key_iv: "75709e7eb18a745527535ce2f98ed580"
server_key_version: "encryptionkey@0"
I20260620 19:09:13.330899 19812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "b0a5d2093aa142b2a942f31a9c682fe1"
format_stamp: "Formatted at 2026-06-20 19:09:13 on dist-test-slave-6wrf"
server_key: "ebab4b148dd23855d0353cbfd2bc53cb"
server_key_iv: "75709e7eb18a745527535ce2f98ed580"
server_key_version: "encryptionkey@0"
I20260620 19:09:13.334971 19812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:09:13.337762 19827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:13.339025 19812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:13.339134 19812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "b0a5d2093aa142b2a942f31a9c682fe1"
format_stamp: "Formatted at 2026-06-20 19:09:13 on dist-test-slave-6wrf"
server_key: "ebab4b148dd23855d0353cbfd2bc53cb"
server_key_iv: "75709e7eb18a745527535ce2f98ed580"
server_key_version: "encryptionkey@0"
I20260620 19:09:13.339294 19812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:13.351696 19812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:13.356227 19812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:13.356477 19812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:13.357148 19812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:13.358091 19812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:13.358165 19812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:13.358232 19812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:13.358284 19812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:13.367643 19812 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:44857
I20260620 19:09:13.367690 19940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:44857 every 8 connection(s)
I20260620 19:09:13.368551 19812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:09:13.371023 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 19812
I20260620 19:09:13.371176 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:09:13.371512 1642 external_mini_cluster.cc:1428] Setting key c181613ea7f8127ffa1f1695f89679e1
Jun 20 19:09:13 dist-test-slave-6wrf krb5kdc[19705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982553, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
I20260620 19:09:13.384526 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.370626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:39743 (local address 127.1.154.190:34921)
0620 19:09:13.370949 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:13.370954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:13.371626 (+ 672us) server_negotiation.cc:407] Connection header received
0620 19:09:13.373233 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:13.373236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:13.373298 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:13.373381 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:13.374756 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.375292 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.376535 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.376795 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.376908 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.379427 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:13.379430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:13.379432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:13.379453 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:13.381691 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:13.382295 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:13.382299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:13.382301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:13.382425 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:13.383044 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:13.383047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:13.383048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:13.383320 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:13.383442 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:13.384072 (+ 630us) server_negotiation.cc:299] Negotiation successful
0620 19:09:13.384229 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260620 19:09:13.385232 19943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.370864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:34921 (local address 127.1.154.129:39743)
0620 19:09:13.371430 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:13.371471 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:13.372990 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:13.373561 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:13.373568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:13.373991 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:13.374577 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.374598 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.375429 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.375433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.376395 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.376405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.376950 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.376954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.377129 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.377815 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:13.377831 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:13.379260 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:13.381866 (+ 2606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:13.381870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:13.381881 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:13.382153 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:13.382701 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:13.382709 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:13.382713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:13.382896 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:13.383462 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:13.383471 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:13.383824 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 19:09:13.383988 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 19:09:13.384251 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":144,"threads_started":1}
I20260620 19:09:13.386715 19941 heartbeater.cc:347] Connected to a master server at 127.1.154.190:34921
I20260620 19:09:13.386979 19941 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:13.387476 19941 heartbeater.cc:511] Master 127.1.154.190:34921 requested a full tablet report, sending...
I20260620 19:09:13.389487 19749 ts_manager.cc:194] Registered new tserver with Master: b0a5d2093aa142b2a942f31a9c682fe1 (127.1.154.129:44857)
I20260620 19:09:13.391037 19749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:39743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:09:13.401015 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:34921
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:13.519225 19946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:13.519479 19946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:13.519521 19946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:13.523567 19946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:13.523636 19946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:13.523694 19946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:09:13.529726 19946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:34921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:13.530764 19946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:13.531774 19946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:13.541524 19951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:13.541507 19952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:13.541508 19954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:13.541857 19946 server_base.cc:1034] running on GCE node
I20260620 19:09:13.542502 19946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:13.543614 19946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:13.544901 19946 hybrid_clock.cc:648] HybridClock initialized: now 1781982553544832 us; error 46 us; skew 500 ppm
Jun 20 19:09:13 dist-test-slave-6wrf krb5kdc[19705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982553, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:13.548348 19946 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:09:13.549283 19946 webserver.cc:460] Webserver started at http://127.1.154.130:40837/ using document root <none> and password file <none>
I20260620 19:09:13.549868 19946 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:13.549928 19946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:13.550087 19946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:13.552021 19946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "921209d8732947c58e61054e4f53108e"
format_stamp: "Formatted at 2026-06-20 19:09:13 on dist-test-slave-6wrf"
server_key: "ef7d8ee9137c4599af79d561f0ea684c"
server_key_iv: "487540db3aa7bac371113c669d880a1f"
server_key_version: "encryptionkey@0"
I20260620 19:09:13.552536 19946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "921209d8732947c58e61054e4f53108e"
format_stamp: "Formatted at 2026-06-20 19:09:13 on dist-test-slave-6wrf"
server_key: "ef7d8ee9137c4599af79d561f0ea684c"
server_key_iv: "487540db3aa7bac371113c669d880a1f"
server_key_version: "encryptionkey@0"
I20260620 19:09:13.556167 19946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:09:13.559024 19961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:13.560331 19946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:13.560474 19946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "921209d8732947c58e61054e4f53108e"
format_stamp: "Formatted at 2026-06-20 19:09:13 on dist-test-slave-6wrf"
server_key: "ef7d8ee9137c4599af79d561f0ea684c"
server_key_iv: "487540db3aa7bac371113c669d880a1f"
server_key_version: "encryptionkey@0"
I20260620 19:09:13.560593 19946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:13.583959 19946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:13.587111 19946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:13.587332 19946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:13.587908 19946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:13.588753 19946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:13.588829 19946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:13.588896 19946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:13.588948 19946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:13.601349 19946 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:45487
I20260620 19:09:13.601369 20074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:45487 every 8 connection(s)
I20260620 19:09:13.602682 19946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:09:13.608700 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 19946
I20260620 19:09:13.608846 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:09:13.609143 1642 external_mini_cluster.cc:1428] Setting key c557a4c339566fb38553ff4bdac04266
Jun 20 19:09:13 dist-test-slave-6wrf krb5kdc[19705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982553, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:13.617914 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.604779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:49577 (local address 127.1.154.190:34921)
0620 19:09:13.604919 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:13.604923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:13.605772 (+ 849us) server_negotiation.cc:407] Connection header received
0620 19:09:13.607041 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:13.607044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:13.607099 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:13.607180 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:13.608343 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.609137 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.610676 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.610900 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.611014 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.614055 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:13.614058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:13.614060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:13.614086 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:13.615630 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:13.616275 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:13.616283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:13.616285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:13.616354 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:13.616709 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:13.616712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:13.616714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:13.616903 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:13.617026 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:13.617549 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 19:09:13.617698 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:09:13.618824 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.605106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:34921 (local address 127.1.154.130:49577)
0620 19:09:13.605620 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:13.605663 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:13.606834 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:13.607353 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:13.607359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:13.607717 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:13.608198 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.608212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.609312 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.609316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.610535 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.610551 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.611064 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.611068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.611252 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.611936 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:13.611952 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:13.613864 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:13.615766 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:13.615772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:13.615789 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:13.616154 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:13.616457 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:13.616460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:13.616462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:13.616605 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:13.617029 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:13.617038 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:13.617305 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 19:09:13.617493 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 19:09:13.617884 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":147,"threads_started":1}
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
I20260620 19:09:13.620433 20075 heartbeater.cc:347] Connected to a master server at 127.1.154.190:34921
I20260620 19:09:13.620726 20075 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:13.621258 20075 heartbeater.cc:511] Master 127.1.154.190:34921 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
I20260620 19:09:13.622990 19749 ts_manager.cc:194] Registered new tserver with Master: 921209d8732947c58e61054e4f53108e (127.1.154.130:45487)
I20260620 19:09:13.623689 19749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:49577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:09:13.641870 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:34921
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:13.751190 20080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:13.751441 20080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:13.751484 20080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:13.755527 20080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:13.755597 20080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:13.755654 20080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:09:13.761369 20080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:34921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:13.763103 20080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:13.764892 20080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:13.776580 20085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:13.776592 20088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:13.776757 20086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:13.777565 20080 server_base.cc:1034] running on GCE node
I20260620 19:09:13.778056 20080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:13.779227 20080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:13.780507 20080 hybrid_clock.cc:648] HybridClock initialized: now 1781982553780462 us; error 92 us; skew 500 ppm
Jun 20 19:09:13 dist-test-slave-6wrf krb5kdc[19705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982553, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:13.783617 20080 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:09:13.784597 20080 webserver.cc:460] Webserver started at http://127.1.154.131:39501/ using document root <none> and password file <none>
I20260620 19:09:13.785120 20080 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:13.785178 20080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:13.785338 20080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:13.787405 20080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "25a852a5ee7846fda1283873a0668e32"
format_stamp: "Formatted at 2026-06-20 19:09:13 on dist-test-slave-6wrf"
server_key: "02860676a6a70c62060ef8488f6c6bac"
server_key_iv: "4fc8b2bdb69cd24833ada87535d35ff2"
server_key_version: "encryptionkey@0"
I20260620 19:09:13.787909 20080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "25a852a5ee7846fda1283873a0668e32"
format_stamp: "Formatted at 2026-06-20 19:09:13 on dist-test-slave-6wrf"
server_key: "02860676a6a70c62060ef8488f6c6bac"
server_key_iv: "4fc8b2bdb69cd24833ada87535d35ff2"
server_key_version: "encryptionkey@0"
I20260620 19:09:13.792126 20080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:09:13.794734 20095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:13.796039 20080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:09:13.796134 20080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "25a852a5ee7846fda1283873a0668e32"
format_stamp: "Formatted at 2026-06-20 19:09:13 on dist-test-slave-6wrf"
server_key: "02860676a6a70c62060ef8488f6c6bac"
server_key_iv: "4fc8b2bdb69cd24833ada87535d35ff2"
server_key_version: "encryptionkey@0"
I20260620 19:09:13.796222 20080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:13.806954 20080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:13.810811 20080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:13.811038 20080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:13.811666 20080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:13.812517 20080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:13.812565 20080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:13.812608 20080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:13.812671 20080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:13.822320 20080 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:39083
I20260620 19:09:13.822343 20208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:39083 every 8 connection(s)
I20260620 19:09:13.823246 20080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:09:13.829201 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 20080
I20260620 19:09:13.829438 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:09:13.829893 1642 external_mini_cluster.cc:1428] Setting key 28ac2c5c8c8d26482c24d262a5464186
Jun 20 19:09:13 dist-test-slave-6wrf krb5kdc[19705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982553, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:13.840708 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.825522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:43103 (local address 127.1.154.190:34921)
0620 19:09:13.825701 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:13.825705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:13.826490 (+ 785us) server_negotiation.cc:407] Connection header received
0620 19:09:13.827701 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:13.827704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:13.827748 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:13.827834 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:13.830000 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.830931 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.833222 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.833445 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.833531 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.836272 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:13.836275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:13.836277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:13.836300 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:13.837917 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:13.838666 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:13.838669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:13.838671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:13.838761 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:13.839302 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:13.839314 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:13.839316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:13.839610 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:13.839723 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:13.840371 (+ 648us) server_negotiation.cc:299] Negotiation successful
0620 19:09:13.840549 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 19:09:13.841506 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.825881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:34921 (local address 127.1.154.131:43103)
0620 19:09:13.826333 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:13.826376 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:13.827508 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:13.828196 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:13.828207 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:13.828753 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:13.829680 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.829711 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.831272 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.831279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.833061 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.833078 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.833584 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.833586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.833811 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.834469 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:13.834485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:13.836091 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:13.838088 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:13.838095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:13.838114 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:13.838514 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:13.838892 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:13.838894 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:13.838897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:13.839073 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:13.839763 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:13.839769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:13.840050 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 19:09:13.840219 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 19:09:13.840493 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":108,"threads_started":1}
I20260620 19:09:13.843080 20209 heartbeater.cc:347] Connected to a master server at 127.1.154.190:34921
I20260620 19:09:13.843472 20209 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:13.844424 20209 heartbeater.cc:511] Master 127.1.154.190:34921 requested a full tablet report, sending...
I20260620 19:09:13.845788 19749 ts_manager.cc:194] Registered new tserver with Master: 25a852a5ee7846fda1283873a0668e32 (127.1.154.131:39083)
I20260620 19:09:13.846474 19749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:43103
I20260620 19:09:13.858373 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:09:13.870565 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.860733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.1.154.190:34921)
0620 19:09:13.861017 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:13.861025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:13.861294 (+ 269us) server_negotiation.cc:407] Connection header received
0620 19:09:13.861542 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:13.861545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:13.861672 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:13.861785 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:13.862653 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.863273 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.864295 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.864686 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.864867 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.866741 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:13.866745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:13.866748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:13.866788 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:13.868217 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:13.868727 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:13.868730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:13.868734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:13.868808 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:13.869132 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:13.869136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:13.869138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:13.869354 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:13.869515 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:13.870132 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 19:09:13.870321 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 19:09:13.874951 19749 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:09:13.878001 19749 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:09:13.894445 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.888318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:44857 (local address 127.0.0.1:52348)
0620 19:09:13.889476 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:13.889492 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:13.889578 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:13.890398 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:13.890401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:13.890417 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:13.890520 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.890526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.891508 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.891511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.892999 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.893008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.894036 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.894039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.894127 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.894161 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 19:09:13.894235 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:09:13.894280 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"mutex_wait_us":656,"thread_start_us":103,"threads_started":1}
I20260620 19:09:13.894586 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.889072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.1.154.130:45487)
0620 19:09:13.889806 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:13.889824 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:13.889843 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:09:13.889901 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:13.889905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:13.890055 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:13.890212 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:13.890555 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.891323 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.893000 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.893655 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.893781 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.893936 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:13.894351 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 19:09:13.894436 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":120,"threads_started":1}
I20260620 19:09:13.894846 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.888318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:45487 (local address 127.0.0.1:51094)
0620 19:09:13.889053 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:13.889104 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:13.889242 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:13.890200 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:13.890206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:13.890227 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:13.890447 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.890456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.891456 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.891459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.892859 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.892869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.893781 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.893784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.893995 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.894040 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:09:13.894145 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 19:09:13.894270 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":144,"spinlock_wait_cycles":12416,"thread_start_us":234,"threads_started":1}
I20260620 19:09:13.894852 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.888668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.1.154.129:44857)
0620 19:09:13.889855 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:13.889861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:13.889879 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:09:13.889941 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:13.889946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:13.890129 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:13.890270 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:13.890642 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.891357 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.893093 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.893921 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.894037 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.894154 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:13.894260 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 19:09:13.894371 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1099,"thread_start_us":101,"threads_started":1}
I20260620 19:09:13.896106 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.888288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:39083 (local address 127.0.0.1:58842)
0620 19:09:13.889055 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:13.889103 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:13.889241 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:13.890284 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:13.890287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:13.890304 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:13.890465 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.890470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.892959 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.892965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.894245 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.894254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.895721 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.895725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.895836 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.895859 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:09:13.895909 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:09:13.895970 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":111,"spinlock_wait_cycles":1920,"thread_start_us":89,"threads_started":1}
I20260620 19:09:13.896766 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.889437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58842 (local address 127.1.154.131:39083)
0620 19:09:13.889833 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:13.889845 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:13.889867 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:09:13.889942 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:13.889946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:13.890129 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:13.890379 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:13.891491 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.892817 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.894373 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.895583 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.896366 (+ 783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.896495 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:13.896542 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 19:09:13.896625 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":207,"threads_started":1}
I20260620 19:09:13.897637 19875 tablet_service.cc:1467] Processing CreateTablet for tablet 0e00059ad14c4ff98d71babfe8eb84b1 (DEFAULT_TABLE table=test-table [id=084c35957eca49c19f5978d1a20df07b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:13.897568 20009 tablet_service.cc:1467] Processing CreateTablet for tablet 0e00059ad14c4ff98d71babfe8eb84b1 (DEFAULT_TABLE table=test-table [id=084c35957eca49c19f5978d1a20df07b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:13.898578 19875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e00059ad14c4ff98d71babfe8eb84b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:13.898605 20009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e00059ad14c4ff98d71babfe8eb84b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:13.899241 20143 tablet_service.cc:1467] Processing CreateTablet for tablet 0e00059ad14c4ff98d71babfe8eb84b1 (DEFAULT_TABLE table=test-table [id=084c35957eca49c19f5978d1a20df07b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:13.900184 20143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e00059ad14c4ff98d71babfe8eb84b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:13.904899 20226 tablet_bootstrap.cc:492] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1: Bootstrap starting.
I20260620 19:09:13.906340 20227 tablet_bootstrap.cc:492] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e: Bootstrap starting.
I20260620 19:09:13.906781 20228 tablet_bootstrap.cc:492] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32: Bootstrap starting.
I20260620 19:09:13.907667 20226 tablet_bootstrap.cc:654] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:13.908318 20227 tablet_bootstrap.cc:654] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:13.908531 20226 log.cc:824] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:13.908696 20228 tablet_bootstrap.cc:654] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:13.909073 20227 log.cc:824] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:13.909826 20228 log.cc:824] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:13.910694 20226 tablet_bootstrap.cc:492] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1: No bootstrap required, opened a new log
I20260620 19:09:13.910904 20226 ts_tablet_manager.cc:1397] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:09:13.911165 20227 tablet_bootstrap.cc:492] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e: No bootstrap required, opened a new log
I20260620 19:09:13.911336 20227 ts_tablet_manager.cc:1397] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:09:13.911579 20228 tablet_bootstrap.cc:492] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32: No bootstrap required, opened a new log
I20260620 19:09:13.911753 20228 ts_tablet_manager.cc:1397] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:09:13.914580 20226 raft_consensus.cc:348] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921209d8732947c58e61054e4f53108e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45487 } } peers { permanent_uuid: "b0a5d2093aa142b2a942f31a9c682fe1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44857 } } peers { permanent_uuid: "25a852a5ee7846fda1283873a0668e32" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 39083 } }
I20260620 19:09:13.914834 20226 raft_consensus.cc:374] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:13.914885 20226 raft_consensus.cc:729] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a5d2093aa142b2a942f31a9c682fe1, State: Initialized, Role: FOLLOWER
I20260620 19:09:13.915323 20226 consensus_queue.cc:260] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921209d8732947c58e61054e4f53108e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45487 } } peers { permanent_uuid: "b0a5d2093aa142b2a942f31a9c682fe1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44857 } } peers { permanent_uuid: "25a852a5ee7846fda1283873a0668e32" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 39083 } }
I20260620 19:09:13.915968 19941 heartbeater.cc:503] Master 127.1.154.190:34921 was elected leader, sending a full tablet report...
I20260620 19:09:13.916596 20228 raft_consensus.cc:348] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921209d8732947c58e61054e4f53108e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45487 } } peers { permanent_uuid: "b0a5d2093aa142b2a942f31a9c682fe1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44857 } } peers { permanent_uuid: "25a852a5ee7846fda1283873a0668e32" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 39083 } }
I20260620 19:09:13.916596 20227 raft_consensus.cc:348] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921209d8732947c58e61054e4f53108e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45487 } } peers { permanent_uuid: "b0a5d2093aa142b2a942f31a9c682fe1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44857 } } peers { permanent_uuid: "25a852a5ee7846fda1283873a0668e32" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 39083 } }
I20260620 19:09:13.917014 20228 raft_consensus.cc:374] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:13.917014 20227 raft_consensus.cc:374] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:13.917085 20227 raft_consensus.cc:729] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 921209d8732947c58e61054e4f53108e, State: Initialized, Role: FOLLOWER
I20260620 19:09:13.917085 20228 raft_consensus.cc:729] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25a852a5ee7846fda1283873a0668e32, State: Initialized, Role: FOLLOWER
I20260620 19:09:13.917786 20228 consensus_queue.cc:260] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921209d8732947c58e61054e4f53108e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45487 } } peers { permanent_uuid: "b0a5d2093aa142b2a942f31a9c682fe1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44857 } } peers { permanent_uuid: "25a852a5ee7846fda1283873a0668e32" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 39083 } }
I20260620 19:09:13.917786 20227 consensus_queue.cc:260] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921209d8732947c58e61054e4f53108e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45487 } } peers { permanent_uuid: "b0a5d2093aa142b2a942f31a9c682fe1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44857 } } peers { permanent_uuid: "25a852a5ee7846fda1283873a0668e32" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 39083 } }
I20260620 19:09:13.918382 20226 ts_tablet_manager.cc:1428] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260620 19:09:13.919164 20228 ts_tablet_manager.cc:1428] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 19:09:13.919227 20227 ts_tablet_manager.cc:1428] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260620 19:09:13.919639 20209 heartbeater.cc:503] Master 127.1.154.190:34921 was elected leader, sending a full tablet report...
I20260620 19:09:13.920523 20075 heartbeater.cc:503] Master 127.1.154.190:34921 was elected leader, sending a full tablet report...
I20260620 19:09:13.980967 20232 raft_consensus.cc:482] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:09:13.981344 20232 raft_consensus.cc:504] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921209d8732947c58e61054e4f53108e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45487 } } peers { permanent_uuid: "b0a5d2093aa142b2a942f31a9c682fe1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44857 } } peers { permanent_uuid: "25a852a5ee7846fda1283873a0668e32" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 39083 } }
I20260620 19:09:13.983253 20232 leader_election.cc:288] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 921209d8732947c58e61054e4f53108e (127.1.154.130:45487), 25a852a5ee7846fda1283873a0668e32 (127.1.154.131:39083)
I20260620 19:09:13.989276 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.983986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:59655 (local address 127.1.154.131:39083)
0620 19:09:13.984164 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:13.984170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:13.984426 (+ 256us) server_negotiation.cc:407] Connection header received
0620 19:09:13.984575 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:13.984578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:13.984638 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:13.984724 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:13.985346 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.986195 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.987518 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.988151 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.988297 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.988342 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:13.989065 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 19:09:13.989141 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 19:09:13.989301 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.983960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:54909 (local address 127.1.154.130:45487)
0620 19:09:13.984097 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:13.984102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:13.984278 (+ 176us) server_negotiation.cc:407] Connection header received
0620 19:09:13.984460 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:13.984464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:13.984558 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:13.984704 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:13.985351 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.986182 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.987564 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.988168 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.988288 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.988342 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:13.988985 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 19:09:13.989134 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:09:13.989285 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.983621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:45487 (local address 127.1.154.129:54909)
0620 19:09:13.984142 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:13.984159 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:13.984305 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:13.984797 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:13.984801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:13.984874 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:13.985121 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.985129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.986326 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.986330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.987385 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.987394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.988453 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.988465 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.988652 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.988734 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 19:09:13.988862 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 19:09:13.989003 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":250,"spinlock_wait_cycles":29312,"thread_start_us":133,"threads_started":1}
I20260620 19:09:13.989285 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:13.983621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:39083 (local address 127.1.154.129:59655)
0620 19:09:13.984321 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:13.984335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:13.984426 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:13.984797 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:13.984801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:13.984848 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:13.985120 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.985129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.986365 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.986369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.987381 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:13.987392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:13.988453 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:13.988465 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:13.988647 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:13.988720 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 19:09:13.988862 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 19:09:13.989003 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":219,"spinlock_wait_cycles":9216,"thread_start_us":182,"threads_started":1}
I20260620 19:09:13.990391 20163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e00059ad14c4ff98d71babfe8eb84b1" candidate_uuid: "b0a5d2093aa142b2a942f31a9c682fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25a852a5ee7846fda1283873a0668e32" is_pre_election: true
I20260620 19:09:13.990371 20029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e00059ad14c4ff98d71babfe8eb84b1" candidate_uuid: "b0a5d2093aa142b2a942f31a9c682fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "921209d8732947c58e61054e4f53108e" is_pre_election: true
I20260620 19:09:13.990732 20163 raft_consensus.cc:2454] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0a5d2093aa142b2a942f31a9c682fe1 in term 0.
I20260620 19:09:13.990868 20029 raft_consensus.cc:2454] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0a5d2093aa142b2a942f31a9c682fe1 in term 0.
I20260620 19:09:13.991308 19830 leader_election.cc:302] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25a852a5ee7846fda1283873a0668e32, b0a5d2093aa142b2a942f31a9c682fe1; no voters:
I20260620 19:09:13.991659 20232 raft_consensus.cc:2781] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:09:13.991750 20232 raft_consensus.cc:482] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:09:13.991802 20232 raft_consensus.cc:3037] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:13.992880 20232 raft_consensus.cc:504] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921209d8732947c58e61054e4f53108e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45487 } } peers { permanent_uuid: "b0a5d2093aa142b2a942f31a9c682fe1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44857 } } peers { permanent_uuid: "25a852a5ee7846fda1283873a0668e32" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 39083 } }
I20260620 19:09:13.993252 20232 leader_election.cc:288] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [CANDIDATE]: Term 1 election: Requested vote from peers 921209d8732947c58e61054e4f53108e (127.1.154.130:45487), 25a852a5ee7846fda1283873a0668e32 (127.1.154.131:39083)
I20260620 19:09:13.993650 20163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e00059ad14c4ff98d71babfe8eb84b1" candidate_uuid: "b0a5d2093aa142b2a942f31a9c682fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25a852a5ee7846fda1283873a0668e32"
I20260620 19:09:13.993724 20029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e00059ad14c4ff98d71babfe8eb84b1" candidate_uuid: "b0a5d2093aa142b2a942f31a9c682fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "921209d8732947c58e61054e4f53108e"
I20260620 19:09:13.993779 20163 raft_consensus.cc:3037] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:13.993885 20029 raft_consensus.cc:3037] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:13.995069 20163 raft_consensus.cc:2454] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a5d2093aa142b2a942f31a9c682fe1 in term 1.
I20260620 19:09:13.995162 20029 raft_consensus.cc:2454] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a5d2093aa142b2a942f31a9c682fe1 in term 1.
I20260620 19:09:13.995460 19829 leader_election.cc:302] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 921209d8732947c58e61054e4f53108e, b0a5d2093aa142b2a942f31a9c682fe1; no voters:
I20260620 19:09:13.995784 20232 raft_consensus.cc:2781] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:13.996174 20232 raft_consensus.cc:686] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [term 1 LEADER]: Becoming Leader. State: Replica: b0a5d2093aa142b2a942f31a9c682fe1, State: Running, Role: LEADER
I20260620 19:09:13.996704 20232 consensus_queue.cc:237] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921209d8732947c58e61054e4f53108e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45487 } } peers { permanent_uuid: "b0a5d2093aa142b2a942f31a9c682fe1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44857 } } peers { permanent_uuid: "25a852a5ee7846fda1283873a0668e32" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 39083 } }
I20260620 19:09:14.001336 19744 catalog_manager.cc:5495] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0a5d2093aa142b2a942f31a9c682fe1 (127.1.154.129). New cstate: current_term: 1 leader_uuid: "b0a5d2093aa142b2a942f31a9c682fe1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921209d8732947c58e61054e4f53108e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0a5d2093aa142b2a942f31a9c682fe1" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 44857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25a852a5ee7846fda1283873a0668e32" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 39083 } health_report { overall_health: UNKNOWN } } }
I20260620 19:09:14.021123 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:14.017406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.1.154.129:44857)
0620 19:09:14.017584 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:14.017587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:14.017668 (+ 81us) server_negotiation.cc:407] Connection header received
0620 19:09:14.017741 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:14.017743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:14.017802 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:14.017919 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:14.018178 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.018740 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.019585 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.019808 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.019912 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:14.020244 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:14.020674 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:14.020829 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:14.020918 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 19:09:14.020990 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 19:09:14.027863 20029 raft_consensus.cc:1261] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e [term 1 FOLLOWER]: Refusing update from remote peer b0a5d2093aa142b2a942f31a9c682fe1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:14.027870 20163 raft_consensus.cc:1261] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32 [term 1 FOLLOWER]: Refusing update from remote peer b0a5d2093aa142b2a942f31a9c682fe1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:14.028589 20237 consensus_queue.cc:1035] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "921209d8732947c58e61054e4f53108e" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:14.028774 20232 consensus_queue.cc:1035] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25a852a5ee7846fda1283873a0668e32" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 39083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:14.039178 20242 mvcc.cc:205] Tried to move back new op lower bound from 7299000541290000384 to 7299000541173514240. Current Snapshot: MvccSnapshot[applied={T|T < 7299000541290000384}]
I20260620 19:09:14.039311 20241 mvcc.cc:205] Tried to move back new op lower bound from 7299000541290000384 to 7299000541173514240. Current Snapshot: MvccSnapshot[applied={T|T < 7299000541290000384}]
I20260620 19:09:14.046865 19750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:09:14.047143 19750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:09:14.049515 19750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 760633c5ea4844ac88c9f3c447ec9eff: Sending DeleteTablet for 3 replicas of tablet 0e00059ad14c4ff98d71babfe8eb84b1
I20260620 19:09:14.050287 19875 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e00059ad14c4ff98d71babfe8eb84b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:14 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:52348
I20260620 19:09:14.050570 20143 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e00059ad14c4ff98d71babfe8eb84b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:14 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:58842
I20260620 19:09:14.050854 20250 tablet_replica.cc:330] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1: stopping tablet replica
I20260620 19:09:14.050819 20009 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e00059ad14c4ff98d71babfe8eb84b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:14 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:51094
I20260620 19:09:14.051162 20250 raft_consensus.cc:2229] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:09:14.051277 20252 tablet_replica.cc:330] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e: stopping tablet replica
I20260620 19:09:14.051538 20252 raft_consensus.cc:2229] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:09:14.051566 20250 raft_consensus.cc:2258] T 0e00059ad14c4ff98d71babfe8eb84b1 P b0a5d2093aa142b2a942f31a9c682fe1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:14.051657 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 19812
I20260620 19:09:14.052459 20252 raft_consensus.cc:2258] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:14.054309 20252 ts_tablet_manager.cc:1905] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:09:14.054617 20251 tablet_replica.cc:330] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32: stopping tablet replica
I20260620 19:09:14.054937 20251 raft_consensus.cc:2229] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:09:14.055234 20251 raft_consensus.cc:2258] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:14.056789 20251 ts_tablet_manager.cc:1905] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:09:14.057192 20252 ts_tablet_manager.cc:1918] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:09:14.057296 20252 log.cc:1196] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/0e00059ad14c4ff98d71babfe8eb84b1
I20260620 19:09:14.057651 20252 ts_tablet_manager.cc:1939] T 0e00059ad14c4ff98d71babfe8eb84b1 P 921209d8732947c58e61054e4f53108e: Deleting consensus metadata
I20260620 19:09:14.058902 19735 catalog_manager.cc:4841] TS 921209d8732947c58e61054e4f53108e (127.1.154.130:45487): tablet 0e00059ad14c4ff98d71babfe8eb84b1 (table test-table [id=084c35957eca49c19f5978d1a20df07b]) successfully deleted
W20260620 19:09:14.059513 19734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:44857 (error 108)
I20260620 19:09:14.060210 20251 ts_tablet_manager.cc:1918] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:09:14.060307 20251 log.cc:1196] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/0e00059ad14c4ff98d71babfe8eb84b1
I20260620 19:09:14.060520 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:14.060252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:44857 (local address 127.0.0.1:52374)
0620 19:09:14.060355 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:14.060428 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:44857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 19:09:14.060614 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 19946
I20260620 19:09:14.060681 20251 ts_tablet_manager.cc:1939] T 0e00059ad14c4ff98d71babfe8eb84b1 P 25a852a5ee7846fda1283873a0668e32: Deleting consensus metadata
W20260620 19:09:14.060755 19734 catalog_manager.cc:4584] TS b0a5d2093aa142b2a942f31a9c682fe1 (127.1.154.129:44857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e00059ad14c4ff98d71babfe8eb84b1: Network error: Client connection negotiation failed: client connection to 127.1.154.129:44857: connect: Connection refused (error 111)
I20260620 19:09:14.067016 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 20080
I20260620 19:09:14.072747 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 19718
2026-06-20T19:09:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2415 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T19:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:14Z Disabled control of system clock
I20260620 19:09:14.105590 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:38089
--webserver_interface=127.1.154.190
--webserver_port=0
--builtin_ntp_servers=127.1.154.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:38089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:09:14.218590 20257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:14.218878 20257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:14.218919 20257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:14.223171 20257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:14.223233 20257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:14.223249 20257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:14.223268 20257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:14.223282 20257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:14.228183 20257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:38089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:38089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:14.229310 20257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:14.230821 20257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:14.239447 20263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:14.239442 20262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:14.239480 20265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:14.239792 20257 server_base.cc:1034] running on GCE node
I20260620 19:09:14.240355 20257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:14.241647 20257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:14.242842 20257 hybrid_clock.cc:648] HybridClock initialized: now 1781982554242823 us; error 32 us; skew 500 ppm
I20260620 19:09:14.245049 20257 webserver.cc:460] Webserver started at http://127.1.154.190:42273/ using document root <none> and password file <none>
I20260620 19:09:14.245724 20257 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:14.245793 20257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:14.246018 20257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:14.248096 20257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "a9be384977384379b5ce9045a8770290"
format_stamp: "Formatted at 2026-06-20 19:09:14 on dist-test-slave-6wrf"
server_key: "9c106f13eaa9bb6ff6f772bed119de7a"
server_key_iv: "8f61b1858b651a409a94f4f616f41f2b"
server_key_version: "encryptionkey@0"
I20260620 19:09:14.248679 20257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "a9be384977384379b5ce9045a8770290"
format_stamp: "Formatted at 2026-06-20 19:09:14 on dist-test-slave-6wrf"
server_key: "9c106f13eaa9bb6ff6f772bed119de7a"
server_key_iv: "8f61b1858b651a409a94f4f616f41f2b"
server_key_version: "encryptionkey@0"
I20260620 19:09:14.253067 20257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 19:09:14.255743 20271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:14.257208 20257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 19:09:14.257318 20257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "a9be384977384379b5ce9045a8770290"
format_stamp: "Formatted at 2026-06-20 19:09:14 on dist-test-slave-6wrf"
server_key: "9c106f13eaa9bb6ff6f772bed119de7a"
server_key_iv: "8f61b1858b651a409a94f4f616f41f2b"
server_key_version: "encryptionkey@0"
I20260620 19:09:14.257477 20257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:14.272454 20257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:14.273344 20257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:14.273589 20257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:14.281283 20257 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:38089
I20260620 19:09:14.281700 20323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:38089 every 8 connection(s)
I20260620 19:09:14.282408 20257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:14.286154 20324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:14.291648 20324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290: Bootstrap starting.
I20260620 19:09:14.291913 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 20257
I20260620 19:09:14.292055 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:14.292362 1642 external_mini_cluster.cc:1428] Setting key b63a4539c0839145dcdd5894fb33f450
I20260620 19:09:14.294426 20324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:14.295157 20324 log.cc:824] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:14.297163 20324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290: No bootstrap required, opened a new log
I20260620 19:09:14.300622 20324 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9be384977384379b5ce9045a8770290" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38089 } }
I20260620 19:09:14.300837 20324 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:14.300900 20324 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9be384977384379b5ce9045a8770290, State: Initialized, Role: FOLLOWER
I20260620 19:09:14.301368 20324 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9be384977384379b5ce9045a8770290" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38089 } }
I20260620 19:09:14.301527 20324 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:14.301638 20324 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:14.301765 20324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:14.302680 20324 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9be384977384379b5ce9045a8770290" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38089 } }
I20260620 19:09:14.303074 20324 leader_election.cc:302] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9be384977384379b5ce9045a8770290; no voters:
I20260620 19:09:14.303454 20324 leader_election.cc:288] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:14.303526 20329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:14.303747 20329 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [term 1 LEADER]: Becoming Leader. State: Replica: a9be384977384379b5ce9045a8770290, State: Running, Role: LEADER
I20260620 19:09:14.304095 20329 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9be384977384379b5ce9045a8770290" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38089 } }
I20260620 19:09:14.304736 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:14.293899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.1.154.190:38089)
0620 19:09:14.294620 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:14.294635 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:14.294670 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:09:14.295369 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:14.295387 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:14.295886 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:14.296263 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:14.296716 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.298250 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.299433 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.300187 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.300385 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:14.301556 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:14.301563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:14.301587 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:14.302251 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:14.302335 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:14.302365 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:14.302488 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:14.302755 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 19:09:14.303366 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":134,"threads_started":1}
I20260620 19:09:14.304843 20324 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:14.305801 20330 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9be384977384379b5ce9045a8770290. Latest consensus state: current_term: 1 leader_uuid: "a9be384977384379b5ce9045a8770290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9be384977384379b5ce9045a8770290" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38089 } } }
I20260620 19:09:14.305924 20330 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:14.306242 20331 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9be384977384379b5ce9045a8770290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9be384977384379b5ce9045a8770290" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38089 } } }
I20260620 19:09:14.306329 20331 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:14.307552 20338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:14.310639 20338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:14.316468 20338 catalog_manager.cc:1347] Generated new cluster ID: 632b2825fc5c4177af6b84da39eeef66
I20260620 19:09:14.316548 20338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:14.323244 20338 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:14.324242 20338 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:14.333056 20338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290: Generated new TSK 0
I20260620 19:09:14.333855 20338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:14.340040 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:38089
--builtin_ntp_servers=127.1.154.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:09:14.452059 20348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:14.452327 20348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:14.452366 20348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:14.456558 20348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:14.456615 20348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:14.456698 20348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:09:14.462582 20348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:38089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:14.464282 20348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:14.466663 20348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:14.476441 20354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:14.476441 20353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:14.476642 20356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:14.476967 20348 server_base.cc:1034] running on GCE node
I20260620 19:09:14.477394 20348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:14.478101 20348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:14.479285 20348 hybrid_clock.cc:648] HybridClock initialized: now 1781982554479282 us; error 48 us; skew 500 ppm
I20260620 19:09:14.481218 20348 webserver.cc:460] Webserver started at http://127.1.154.129:34891/ using document root <none> and password file <none>
I20260620 19:09:14.481894 20348 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:14.481963 20348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:14.482169 20348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:14.484234 20348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "73e6a5462dff49a4bcb8581350a46c34"
format_stamp: "Formatted at 2026-06-20 19:09:14 on dist-test-slave-6wrf"
server_key: "545c90dadfaf0b0aa2ad26ce4c95ee34"
server_key_iv: "e51e8a89852c8a890aa2ff121188fb64"
server_key_version: "encryptionkey@0"
I20260620 19:09:14.484795 20348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "73e6a5462dff49a4bcb8581350a46c34"
format_stamp: "Formatted at 2026-06-20 19:09:14 on dist-test-slave-6wrf"
server_key: "545c90dadfaf0b0aa2ad26ce4c95ee34"
server_key_iv: "e51e8a89852c8a890aa2ff121188fb64"
server_key_version: "encryptionkey@0"
I20260620 19:09:14.488485 20348 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 19:09:14.490957 20362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:14.492502 20348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:09:14.492652 20348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "73e6a5462dff49a4bcb8581350a46c34"
format_stamp: "Formatted at 2026-06-20 19:09:14 on dist-test-slave-6wrf"
server_key: "545c90dadfaf0b0aa2ad26ce4c95ee34"
server_key_iv: "e51e8a89852c8a890aa2ff121188fb64"
server_key_version: "encryptionkey@0"
I20260620 19:09:14.492767 20348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:14.525487 20348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:14.526232 20348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:14.526433 20348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:14.527048 20348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:14.527985 20348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:14.528059 20348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:14.528132 20348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:14.528177 20348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:14.537685 20348 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:45655
I20260620 19:09:14.537811 20475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:45655 every 8 connection(s)
I20260620 19:09:14.538683 20348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:09:14.547123 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 20348
I20260620 19:09:14.547297 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:09:14.547581 1642 external_mini_cluster.cc:1428] Setting key 7e76baf0f585212088870ce466bfc41e
I20260620 19:09:14.549508 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:14.540672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:41721 (local address 127.1.154.190:38089)
0620 19:09:14.540823 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:14.540827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:14.542447 (+ 1620us) server_negotiation.cc:407] Connection header received
0620 19:09:14.543984 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:14.543987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:14.544068 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:14.544176 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:14.545069 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.545690 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.546824 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.547046 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.547158 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:14.548932 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:14.548935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:14.548937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:14.548948 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:14.548970 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:14.548978 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:14.549045 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:14.549252 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 19:09:14.549344 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 19:09:14.550249 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:38089
--builtin_ntp_servers=127.1.154.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:09:14.551169 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:14.541066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:38089 (local address 127.1.154.129:41721)
0620 19:09:14.542258 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:14.542303 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:14.543665 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:14.544368 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:14.544375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:14.544457 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:14.544926 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:14.544937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.545813 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.545817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:14.546678 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:14.546687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.547197 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.547201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:14.547446 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:14.548463 (+ 1017us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:09:14.548467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:14.548657 (+ 190us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:09:14.548783 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:14.549074 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:14.549083 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:14.549089 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:09:14.549222 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 19:09:14.549554 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":118,"threads_started":1}
I20260620 19:09:14.552469 20476 heartbeater.cc:347] Connected to a master server at 127.1.154.190:38089
I20260620 19:09:14.552722 20476 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:14.553236 20476 heartbeater.cc:511] Master 127.1.154.190:38089 requested a full tablet report, sending...
I20260620 19:09:14.554807 20288 ts_manager.cc:194] Registered new tserver with Master: 73e6a5462dff49a4bcb8581350a46c34 (127.1.154.129:45655)
I20260620 19:09:14.555953 20288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.154.129:41721
W20260620 19:09:14.666814 20479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:14.667152 20479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:14.667243 20479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:14.672178 20479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:14.672339 20479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:14.672468 20479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:09:14.679198 20479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:38089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:14.680418 20479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:14.681910 20479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:14.693634 20485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:14.693820 20479 server_base.cc:1034] running on GCE node
W20260620 19:09:14.693631 20487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:14.693629 20484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:14.694335 20479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:14.695082 20479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:14.696273 20479 hybrid_clock.cc:648] HybridClock initialized: now 1781982554696262 us; error 26 us; skew 500 ppm
I20260620 19:09:14.698560 20479 webserver.cc:460] Webserver started at http://127.1.154.130:46367/ using document root <none> and password file <none>
I20260620 19:09:14.699168 20479 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:14.699262 20479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:14.699504 20479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:14.701560 20479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "36b3d2ee662f4f58ab17377f7737f1c7"
format_stamp: "Formatted at 2026-06-20 19:09:14 on dist-test-slave-6wrf"
server_key: "194ae7ef22c1184b3f9edcb29e526002"
server_key_iv: "5434ccdaa429e56832ecb51d2e5f496d"
server_key_version: "encryptionkey@0"
I20260620 19:09:14.702170 20479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "36b3d2ee662f4f58ab17377f7737f1c7"
format_stamp: "Formatted at 2026-06-20 19:09:14 on dist-test-slave-6wrf"
server_key: "194ae7ef22c1184b3f9edcb29e526002"
server_key_iv: "5434ccdaa429e56832ecb51d2e5f496d"
server_key_version: "encryptionkey@0"
I20260620 19:09:14.705976 20479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 19:09:14.708490 20493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:14.710700 20479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:14.710927 20479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "36b3d2ee662f4f58ab17377f7737f1c7"
format_stamp: "Formatted at 2026-06-20 19:09:14 on dist-test-slave-6wrf"
server_key: "194ae7ef22c1184b3f9edcb29e526002"
server_key_iv: "5434ccdaa429e56832ecb51d2e5f496d"
server_key_version: "encryptionkey@0"
I20260620 19:09:14.711210 20479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:14.722348 20479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:14.723104 20479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:14.723312 20479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:14.723928 20479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:14.725471 20479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:14.725605 20479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:14.725726 20479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:14.725773 20479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:14.736202 20479 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:39637
I20260620 19:09:14.736219 20606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:39637 every 8 connection(s)
I20260620 19:09:14.737151 20479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:09:14.747509 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 20479
I20260620 19:09:14.747889 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:09:14.748306 1642 external_mini_cluster.cc:1428] Setting key 3360cdc508eb326115b4f698b4784a28
I20260620 19:09:14.749475 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:14.739111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:46399 (local address 127.1.154.190:38089)
0620 19:09:14.739315 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:14.739319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:14.740790 (+ 1471us) server_negotiation.cc:407] Connection header received
0620 19:09:14.742811 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:14.742816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:14.742910 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:14.743034 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:14.744403 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.745055 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.746619 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.746890 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.747027 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:14.748797 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:14.748802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:14.748805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:14.748847 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:14.748884 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:14.748899 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:14.749006 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:14.749223 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 19:09:14.749320 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 19:09:14.750768 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:14.739496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:38089 (local address 127.1.154.130:46399)
0620 19:09:14.740506 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:14.740576 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:14.742442 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:14.743279 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:14.743288 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:14.743410 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:14.744187 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:14.744204 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.745218 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.745223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:14.746436 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:14.746447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.747089 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.747094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:14.747305 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:14.748292 (+ 987us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:09:14.748295 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:14.748387 (+ 92us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:09:14.748523 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:14.749054 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:14.749063 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:14.749068 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:09:14.749214 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 19:09:14.749587 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":117,"threads_started":1}
I20260620 19:09:14.751070 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:38089
--builtin_ntp_servers=127.1.154.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:09:14.752153 20607 heartbeater.cc:347] Connected to a master server at 127.1.154.190:38089
I20260620 19:09:14.752513 20607 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:14.753275 20607 heartbeater.cc:511] Master 127.1.154.190:38089 requested a full tablet report, sending...
I20260620 19:09:14.754586 20288 ts_manager.cc:194] Registered new tserver with Master: 36b3d2ee662f4f58ab17377f7737f1c7 (127.1.154.130:39637)
I20260620 19:09:14.755553 20288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.154.130:46399
W20260620 19:09:14.878448 20610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:14.878710 20610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:14.878749 20610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:14.882742 20610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:14.882800 20610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:14.882860 20610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:09:14.887631 20610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:38089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:14.888816 20610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:14.890290 20610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:14.903586 20615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:14.903611 20618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:14.903807 20616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:14.904050 20610 server_base.cc:1034] running on GCE node
I20260620 19:09:14.904551 20610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:14.905225 20610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:14.906535 20610 hybrid_clock.cc:648] HybridClock initialized: now 1781982554906507 us; error 51 us; skew 500 ppm
I20260620 19:09:14.908442 20610 webserver.cc:460] Webserver started at http://127.1.154.131:37001/ using document root <none> and password file <none>
I20260620 19:09:14.909056 20610 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:14.909148 20610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:14.909355 20610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:14.911509 20610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "1db09a762815477bb7bf2abaf6cab556"
format_stamp: "Formatted at 2026-06-20 19:09:14 on dist-test-slave-6wrf"
server_key: "b5028ebbe6490824517992f12dd230e4"
server_key_iv: "cbca82b1ef1c851ca753a3519f8a8626"
server_key_version: "encryptionkey@0"
I20260620 19:09:14.912106 20610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "1db09a762815477bb7bf2abaf6cab556"
format_stamp: "Formatted at 2026-06-20 19:09:14 on dist-test-slave-6wrf"
server_key: "b5028ebbe6490824517992f12dd230e4"
server_key_iv: "cbca82b1ef1c851ca753a3519f8a8626"
server_key_version: "encryptionkey@0"
I20260620 19:09:14.915916 20610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:09:14.918457 20624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:14.919799 20610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:09:14.919901 20610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "1db09a762815477bb7bf2abaf6cab556"
format_stamp: "Formatted at 2026-06-20 19:09:14 on dist-test-slave-6wrf"
server_key: "b5028ebbe6490824517992f12dd230e4"
server_key_iv: "cbca82b1ef1c851ca753a3519f8a8626"
server_key_version: "encryptionkey@0"
I20260620 19:09:14.920014 20610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:14.935791 20610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:14.936558 20610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:14.936781 20610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:14.937409 20610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:14.938575 20610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:14.938650 20610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:14.938719 20610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:14.938768 20610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:14.948565 20610 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:45901
I20260620 19:09:14.948593 20737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:45901 every 8 connection(s)
I20260620 19:09:14.949527 20610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:09:14.959712 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 20610
I20260620 19:09:14.959909 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:09:14.960078 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:14.952277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:33565 (local address 127.1.154.190:38089)
0620 19:09:14.952523 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:14.952528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:14.953634 (+ 1106us) server_negotiation.cc:407] Connection header received
0620 19:09:14.955066 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:14.955069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:14.955133 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:14.955238 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:14.956126 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.956663 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.957866 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.958086 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.958198 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:14.959569 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:14.959572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:14.959574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:14.959586 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:14.959602 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:14.959610 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:14.959689 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:14.959865 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 19:09:14.959924 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 19:09:14.960450 1642 external_mini_cluster.cc:1428] Setting key 9f28a491cc63220e7b53b8db07f81ace
I20260620 19:09:14.960968 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:14.952716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:38089 (local address 127.1.154.131:33565)
0620 19:09:14.953428 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:14.953470 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:14.954836 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:14.955405 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:14.955412 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:14.955494 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:14.955972 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:14.955985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.956796 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.956807 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:14.957735 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:14.957744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.958246 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.958251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:14.958401 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:14.959286 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:09:14.959288 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:14.959325 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:09:14.959394 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:14.959703 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:14.959714 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:14.959725 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 19:09:14.959845 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 19:09:14.960113 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":133,"threads_started":1}
I20260620 19:09:14.962492 20738 heartbeater.cc:347] Connected to a master server at 127.1.154.190:38089
I20260620 19:09:14.962735 20738 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:14.963263 20738 heartbeater.cc:511] Master 127.1.154.190:38089 requested a full tablet report, sending...
I20260620 19:09:14.964521 20288 ts_manager.cc:194] Registered new tserver with Master: 1db09a762815477bb7bf2abaf6cab556 (127.1.154.131:45901)
I20260620 19:09:14.965409 20288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.154.131:33565
I20260620 19:09:14.975776 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:09:14.983351 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:14.977955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.1.154.190:38089)
0620 19:09:14.978267 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:14.978273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:14.978543 (+ 270us) server_negotiation.cc:407] Connection header received
0620 19:09:14.978773 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:14.978776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:14.978842 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:14.978943 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:14.979300 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.980002 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.981015 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:14.981362 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:14.981553 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:14.982626 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:14.982630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:14.982633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:14.982650 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:14.982688 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:14.982702 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:14.982879 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:14.983077 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 19:09:14.983132 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260620 19:09:14.986647 20288 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:09:14.989852 20288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:09:15.006618 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:15.000466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44046 (local address 127.1.154.129:45655)
0620 19:09:15.001741 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:15.001749 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:15.001787 (+ 38us) server_negotiation.cc:407] Connection header received
0620 19:09:15.001859 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:15.001864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:15.002097 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:15.002261 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:15.002503 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.003661 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.005127 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.005960 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.006077 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:15.006239 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:15.006331 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 19:09:15.006428 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1171,"thread_start_us":138,"threads_started":1}
I20260620 19:09:15.006618 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:14.999662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.1.154.131:45901)
0620 19:09:15.000170 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:15.000188 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:15.000648 (+ 460us) server_negotiation.cc:407] Connection header received
0620 19:09:15.000991 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:15.000998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:15.001299 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:15.001460 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:15.002118 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.003205 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.004771 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.005452 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.005575 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:15.005951 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:15.006353 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 19:09:15.006454 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":130,"threads_started":1}
I20260620 19:09:15.006618 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:14.999985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:45655 (local address 127.0.0.1:44046)
0620 19:09:15.000587 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:15.000606 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:15.000748 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:15.002268 (+ 1520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:15.002271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:15.002291 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:15.002375 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:15.002381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.003788 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.003791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:15.004994 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:15.005004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.006093 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.006097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:15.006185 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:15.006223 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 19:09:15.006303 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 19:09:15.006435 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":1,"thread_start_us":190,"threads_started":1}
I20260620 19:09:15.007045 20751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:14.999557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:45901 (local address 127.0.0.1:40198)
0620 19:09:15.000463 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:15.000525 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:15.000753 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:15.001698 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:15.001705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:15.001732 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:15.001936 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:15.001944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.003318 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.003323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:15.004640 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:15.004654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.006025 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.006031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:15.006148 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:15.006186 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 19:09:15.006785 (+ 599us) client_negotiation.cc:240] Negotiation successful
0620 19:09:15.006913 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":24,"thread_start_us":78,"threads_started":1}
I20260620 19:09:15.007686 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:15.001024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.1.154.130:39637)
0620 19:09:15.002230 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:15.002241 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:15.002261 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:09:15.002327 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:15.002331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:15.002519 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:15.002670 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:15.002906 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.003954 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.005352 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.006172 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.006287 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:15.007462 (+ 1175us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:15.007503 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 19:09:15.007568 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1101,"thread_start_us":140,"threads_started":1}
I20260620 19:09:15.007953 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:15.000770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:39637 (local address 127.0.0.1:44010)
0620 19:09:15.001142 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:15.001157 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:15.001403 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:15.002667 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:15.002670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:15.002688 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:15.002771 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:15.002776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.004087 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.004090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:15.005233 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:15.005263 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.006419 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.006422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:15.006495 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:15.006516 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:09:15.006574 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 19:09:15.006632 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":1,"thread_start_us":116,"threads_started":1}
I20260620 19:09:15.010067 20672 tablet_service.cc:1467] Processing CreateTablet for tablet 4a40a03a169b4d17940e3b1a1aa27265 (DEFAULT_TABLE table=test-table [id=69f841d40ea74351be412ebf128cba22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:15.010067 20410 tablet_service.cc:1467] Processing CreateTablet for tablet 4a40a03a169b4d17940e3b1a1aa27265 (DEFAULT_TABLE table=test-table [id=69f841d40ea74351be412ebf128cba22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:15.011194 20672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a40a03a169b4d17940e3b1a1aa27265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:15.011194 20410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a40a03a169b4d17940e3b1a1aa27265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:15.013247 20541 tablet_service.cc:1467] Processing CreateTablet for tablet 4a40a03a169b4d17940e3b1a1aa27265 (DEFAULT_TABLE table=test-table [id=69f841d40ea74351be412ebf128cba22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:15.014212 20541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a40a03a169b4d17940e3b1a1aa27265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:15.018579 20756 tablet_bootstrap.cc:492] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34: Bootstrap starting.
I20260620 19:09:15.018991 20755 tablet_bootstrap.cc:492] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556: Bootstrap starting.
I20260620 19:09:15.020515 20757 tablet_bootstrap.cc:492] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7: Bootstrap starting.
I20260620 19:09:15.022444 20757 tablet_bootstrap.cc:654] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:15.022617 20756 tablet_bootstrap.cc:654] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:15.023254 20757 log.cc:824] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:15.023339 20756 log.cc:824] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:15.024554 20755 tablet_bootstrap.cc:654] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:15.025274 20757 tablet_bootstrap.cc:492] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7: No bootstrap required, opened a new log
I20260620 19:09:15.025321 20755 log.cc:824] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:15.025382 20756 tablet_bootstrap.cc:492] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34: No bootstrap required, opened a new log
I20260620 19:09:15.025481 20757 ts_tablet_manager.cc:1397] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:09:15.025543 20756 ts_tablet_manager.cc:1397] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260620 19:09:15.027484 20755 tablet_bootstrap.cc:492] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556: No bootstrap required, opened a new log
I20260620 19:09:15.027669 20755 ts_tablet_manager.cc:1397] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260620 19:09:15.030097 20756 raft_consensus.cc:348] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db09a762815477bb7bf2abaf6cab556" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45901 } } peers { permanent_uuid: "73e6a5462dff49a4bcb8581350a46c34" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45655 } } peers { permanent_uuid: "36b3d2ee662f4f58ab17377f7737f1c7" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39637 } }
I20260620 19:09:15.030040 20757 raft_consensus.cc:348] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db09a762815477bb7bf2abaf6cab556" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45901 } } peers { permanent_uuid: "73e6a5462dff49a4bcb8581350a46c34" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45655 } } peers { permanent_uuid: "36b3d2ee662f4f58ab17377f7737f1c7" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39637 } }
I20260620 19:09:15.030376 20756 raft_consensus.cc:374] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:15.030409 20757 raft_consensus.cc:374] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:15.030426 20756 raft_consensus.cc:729] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e6a5462dff49a4bcb8581350a46c34, State: Initialized, Role: FOLLOWER
I20260620 19:09:15.030471 20757 raft_consensus.cc:729] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b3d2ee662f4f58ab17377f7737f1c7, State: Initialized, Role: FOLLOWER
I20260620 19:09:15.030887 20756 consensus_queue.cc:260] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db09a762815477bb7bf2abaf6cab556" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45901 } } peers { permanent_uuid: "73e6a5462dff49a4bcb8581350a46c34" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45655 } } peers { permanent_uuid: "36b3d2ee662f4f58ab17377f7737f1c7" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39637 } }
I20260620 19:09:15.031013 20757 consensus_queue.cc:260] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db09a762815477bb7bf2abaf6cab556" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45901 } } peers { permanent_uuid: "73e6a5462dff49a4bcb8581350a46c34" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45655 } } peers { permanent_uuid: "36b3d2ee662f4f58ab17377f7737f1c7" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39637 } }
I20260620 19:09:15.031786 20607 heartbeater.cc:503] Master 127.1.154.190:38089 was elected leader, sending a full tablet report...
I20260620 19:09:15.031842 20755 raft_consensus.cc:348] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db09a762815477bb7bf2abaf6cab556" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45901 } } peers { permanent_uuid: "73e6a5462dff49a4bcb8581350a46c34" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45655 } } peers { permanent_uuid: "36b3d2ee662f4f58ab17377f7737f1c7" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39637 } }
I20260620 19:09:15.032073 20755 raft_consensus.cc:374] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:15.032083 20757 ts_tablet_manager.cc:1428] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:09:15.032127 20755 raft_consensus.cc:729] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db09a762815477bb7bf2abaf6cab556, State: Initialized, Role: FOLLOWER
I20260620 19:09:15.032519 20755 consensus_queue.cc:260] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db09a762815477bb7bf2abaf6cab556" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45901 } } peers { permanent_uuid: "73e6a5462dff49a4bcb8581350a46c34" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45655 } } peers { permanent_uuid: "36b3d2ee662f4f58ab17377f7737f1c7" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39637 } }
I20260620 19:09:15.033017 20756 ts_tablet_manager.cc:1428] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 19:09:15.033293 20738 heartbeater.cc:503] Master 127.1.154.190:38089 was elected leader, sending a full tablet report...
I20260620 19:09:15.033955 20755 ts_tablet_manager.cc:1428] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260620 19:09:15.034173 20476 heartbeater.cc:503] Master 127.1.154.190:38089 was elected leader, sending a full tablet report...
W20260620 19:09:15.039953 20477 tablet.cc:2369] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:09:15.063223 20763 raft_consensus.cc:482] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:09:15.063442 20763 raft_consensus.cc:504] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db09a762815477bb7bf2abaf6cab556" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45901 } } peers { permanent_uuid: "73e6a5462dff49a4bcb8581350a46c34" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45655 } } peers { permanent_uuid: "36b3d2ee662f4f58ab17377f7737f1c7" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39637 } }
I20260620 19:09:15.064632 20763 leader_election.cc:288] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73e6a5462dff49a4bcb8581350a46c34 (127.1.154.129:45655), 36b3d2ee662f4f58ab17377f7737f1c7 (127.1.154.130:39637)
I20260620 19:09:15.069765 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:15.064845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:45655 (local address 127.1.154.131:37273)
0620 19:09:15.065041 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:15.065059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:15.065150 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:15.065514 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:15.065518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:15.065537 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:15.065683 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:15.065695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.066758 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.066760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:15.067976 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:15.067986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.069034 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.069040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:15.069349 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:15.069413 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 19:09:15.069523 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 19:09:15.069578 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":69}
I20260620 19:09:15.069782 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:15.065010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:37273 (local address 127.1.154.129:45655)
0620 19:09:15.065150 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:15.065155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:15.065187 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:09:15.065253 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:15.065256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:15.065307 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:15.065385 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:15.065882 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.066622 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.068122 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.068745 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.068893 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:15.068964 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:15.069548 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 19:09:15.069665 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 19:09:15.069785 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:15.065419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:37113 (local address 127.1.154.130:39637)
0620 19:09:15.065537 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:15.065541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:15.065665 (+ 124us) server_negotiation.cc:407] Connection header received
0620 19:09:15.065815 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:15.065818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:15.065885 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:15.065958 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:15.066250 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.066888 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.068259 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.068870 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.068979 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:15.069026 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:15.069548 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 19:09:15.069658 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:09:15.069783 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:15.065132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:39637 (local address 127.1.154.131:37113)
0620 19:09:15.065544 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:15.065559 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:15.065704 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:15.066013 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:15.066016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:15.066033 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:15.066136 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:15.066141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.066997 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.067000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:15.068127 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:15.068134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.069034 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.069040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:15.069353 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:15.069404 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 19:09:15.069502 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 19:09:15.069578 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":37,"thread_start_us":131,"threads_started":1}
I20260620 19:09:15.070463 20430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a40a03a169b4d17940e3b1a1aa27265" candidate_uuid: "1db09a762815477bb7bf2abaf6cab556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73e6a5462dff49a4bcb8581350a46c34" is_pre_election: true
I20260620 19:09:15.070472 20561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a40a03a169b4d17940e3b1a1aa27265" candidate_uuid: "1db09a762815477bb7bf2abaf6cab556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b3d2ee662f4f58ab17377f7737f1c7" is_pre_election: true
I20260620 19:09:15.070770 20430 raft_consensus.cc:2454] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1db09a762815477bb7bf2abaf6cab556 in term 0.
I20260620 19:09:15.070770 20561 raft_consensus.cc:2454] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1db09a762815477bb7bf2abaf6cab556 in term 0.
I20260620 19:09:15.071286 20627 leader_election.cc:302] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1db09a762815477bb7bf2abaf6cab556, 73e6a5462dff49a4bcb8581350a46c34; no voters:
I20260620 19:09:15.071599 20763 raft_consensus.cc:2781] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:09:15.071681 20763 raft_consensus.cc:482] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:09:15.071712 20763 raft_consensus.cc:3037] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:15.072790 20763 raft_consensus.cc:504] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db09a762815477bb7bf2abaf6cab556" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45901 } } peers { permanent_uuid: "73e6a5462dff49a4bcb8581350a46c34" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45655 } } peers { permanent_uuid: "36b3d2ee662f4f58ab17377f7737f1c7" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39637 } }
I20260620 19:09:15.073133 20763 leader_election.cc:288] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [CANDIDATE]: Term 1 election: Requested vote from peers 73e6a5462dff49a4bcb8581350a46c34 (127.1.154.129:45655), 36b3d2ee662f4f58ab17377f7737f1c7 (127.1.154.130:39637)
I20260620 19:09:15.073559 20430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a40a03a169b4d17940e3b1a1aa27265" candidate_uuid: "1db09a762815477bb7bf2abaf6cab556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73e6a5462dff49a4bcb8581350a46c34"
I20260620 19:09:15.073652 20561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a40a03a169b4d17940e3b1a1aa27265" candidate_uuid: "1db09a762815477bb7bf2abaf6cab556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b3d2ee662f4f58ab17377f7737f1c7"
I20260620 19:09:15.073763 20430 raft_consensus.cc:3037] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:15.073772 20561 raft_consensus.cc:3037] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:15.074744 20561 raft_consensus.cc:2454] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1db09a762815477bb7bf2abaf6cab556 in term 1.
I20260620 19:09:15.074848 20430 raft_consensus.cc:2454] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1db09a762815477bb7bf2abaf6cab556 in term 1.
I20260620 19:09:15.075047 20627 leader_election.cc:302] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1db09a762815477bb7bf2abaf6cab556, 36b3d2ee662f4f58ab17377f7737f1c7; no voters:
I20260620 19:09:15.075242 20763 raft_consensus.cc:2781] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:15.075512 20763 raft_consensus.cc:686] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [term 1 LEADER]: Becoming Leader. State: Replica: 1db09a762815477bb7bf2abaf6cab556, State: Running, Role: LEADER
I20260620 19:09:15.075872 20763 consensus_queue.cc:237] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db09a762815477bb7bf2abaf6cab556" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45901 } } peers { permanent_uuid: "73e6a5462dff49a4bcb8581350a46c34" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45655 } } peers { permanent_uuid: "36b3d2ee662f4f58ab17377f7737f1c7" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39637 } }
I20260620 19:09:15.079085 20288 catalog_manager.cc:5495] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1db09a762815477bb7bf2abaf6cab556 (127.1.154.131). New cstate: current_term: 1 leader_uuid: "1db09a762815477bb7bf2abaf6cab556" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db09a762815477bb7bf2abaf6cab556" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 45901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73e6a5462dff49a4bcb8581350a46c34" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36b3d2ee662f4f58ab17377f7737f1c7" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39637 } health_report { overall_health: UNKNOWN } } }
I20260620 19:09:15.113067 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:15.108525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40214 (local address 127.1.154.131:45901)
0620 19:09:15.108698 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:15.108704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:15.108724 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:09:15.109011 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:15.109014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:15.109075 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:15.109180 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:15.109618 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.110133 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.111069 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:15.111486 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:15.111582 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:15.112045 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:15.112615 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:15.112741 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:15.112901 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 19:09:15.112934 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 19:09:15.119264 20430 raft_consensus.cc:1261] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34 [term 1 FOLLOWER]: Refusing update from remote peer 1db09a762815477bb7bf2abaf6cab556: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:15.119453 20561 raft_consensus.cc:1261] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7 [term 1 FOLLOWER]: Refusing update from remote peer 1db09a762815477bb7bf2abaf6cab556: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:15.120157 20765 consensus_queue.cc:1035] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73e6a5462dff49a4bcb8581350a46c34" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 45655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:15.120376 20763 consensus_queue.cc:1035] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b3d2ee662f4f58ab17377f7737f1c7" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 39637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:15.128829 20769 mvcc.cc:205] Tried to move back new op lower bound from 7299000545759252480 to 7299000545593573376. Current Snapshot: MvccSnapshot[applied={T|T < 7299000545759252480}]
I20260620 19:09:15.137480 20288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:09:15.137715 20288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:09:15.140059 20288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9be384977384379b5ce9045a8770290: Sending DeleteTablet for 3 replicas of tablet 4a40a03a169b4d17940e3b1a1aa27265
I20260620 19:09:15.140830 20672 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a40a03a169b4d17940e3b1a1aa27265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:15 UTC) from {username='slave'} at 127.0.0.1:40198
I20260620 19:09:15.140960 20541 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a40a03a169b4d17940e3b1a1aa27265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:15 UTC) from {username='slave'} at 127.0.0.1:44010
I20260620 19:09:15.141340 20778 tablet_replica.cc:330] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556: stopping tablet replica
I20260620 19:09:15.141327 20410 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a40a03a169b4d17940e3b1a1aa27265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:15 UTC) from {username='slave'} at 127.0.0.1:44046
I20260620 19:09:15.141499 20779 tablet_replica.cc:330] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7: stopping tablet replica
I20260620 19:09:15.141686 20778 raft_consensus.cc:2229] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:09:15.141741 20779 raft_consensus.cc:2229] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:09:15.142040 20778 raft_consensus.cc:2258] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:15.142023 20779 raft_consensus.cc:2258] T 4a40a03a169b4d17940e3b1a1aa27265 P 36b3d2ee662f4f58ab17377f7737f1c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:15.142346 20780 tablet_replica.cc:330] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34: stopping tablet replica
I20260620 19:09:15.142688 20780 raft_consensus.cc:2229] T 4a40a03a169b4d17940e3b1a1aa27265 P 73e6a5462dff49a4bcb8581350a46c34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:09:15.142717 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 20348
I20260620 19:09:15.146620 20778 ts_tablet_manager.cc:1905] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:09:15.149302 20778 ts_tablet_manager.cc:1918] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:09:15.149402 20778 log.cc:1196] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/4a40a03a169b4d17940e3b1a1aa27265
W20260620 19:09:15.149441 20274 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:45655 (error 108)
I20260620 19:09:15.149747 20778 ts_tablet_manager.cc:1939] T 4a40a03a169b4d17940e3b1a1aa27265 P 1db09a762815477bb7bf2abaf6cab556: Deleting consensus metadata
I20260620 19:09:15.150250 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 20479
I20260620 19:09:15.151507 20273 catalog_manager.cc:4841] TS 1db09a762815477bb7bf2abaf6cab556 (127.1.154.131:45901): tablet 4a40a03a169b4d17940e3b1a1aa27265 (table test-table [id=69f841d40ea74351be412ebf128cba22]) successfully deleted
I20260620 19:09:15.152653 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:15.152186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:45655 (local address 127.0.0.1:44062)
0620 19:09:15.152498 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:15.152566 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:45655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
W20260620 19:09:15.152889 20274 catalog_manager.cc:4584] TS 73e6a5462dff49a4bcb8581350a46c34 (127.1.154.129:45655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a40a03a169b4d17940e3b1a1aa27265: Network error: Client connection negotiation failed: client connection to 127.1.154.129:45655: connect: Connection refused (error 111)
I20260620 19:09:15.156631 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 20610
I20260620 19:09:15.158257 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:15.156786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:39637 (local address 127.0.0.1:44026)
0620 19:09:15.158122 (+ 1336us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:15.158178 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.130:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1250}
I20260620 19:09:15.162533 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 20257
2026-06-20T19:09:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1088 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:15 dist-test-slave-6wrf krb5kdc[20784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:15 dist-test-slave-6wrf krb5kdc[20784](info): set up 2 sockets
Jun 20 19:09:15 dist-test-slave-6wrf krb5kdc[20784](info): commencing operation
krb5kdc: starting...
W20260620 19:09:16.229526 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:16 dist-test-slave-6wrf krb5kdc[20784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:09:16.356314 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:40229
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:40229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:16.474076 20797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:16.474354 20797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:16.474397 20797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:16.478595 20797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:16.478663 20797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:16.478683 20797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:16.478701 20797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:16.478716 20797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:16.484338 20797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:40229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:40229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:16.485519 20797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:16.486608 20797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:16.496102 20803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:16.496102 20805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:16.496345 20797 server_base.cc:1034] running on GCE node
W20260620 19:09:16.496127 20802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:16.497046 20797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:16.498210 20797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:16.499549 20797 hybrid_clock.cc:648] HybridClock initialized: now 1781982556499463 us; error 41 us; skew 500 ppm
Jun 20 19:09:16 dist-test-slave-6wrf krb5kdc[20784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982556, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:16.503645 20797 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:09:16.504657 20797 webserver.cc:460] Webserver started at http://127.1.154.190:42975/ using document root <none> and password file <none>
I20260620 19:09:16.505272 20797 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:16.505339 20797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:16.505553 20797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:16.507587 20797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "c3f29237f61e4691b254f27720d56f05"
format_stamp: "Formatted at 2026-06-20 19:09:16 on dist-test-slave-6wrf"
server_key: "febc819fd99ffbdae6165c76569693a3"
server_key_iv: "4860b1cfc162548b6f04777187e70574"
server_key_version: "encryptionkey@0"
I20260620 19:09:16.508155 20797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "c3f29237f61e4691b254f27720d56f05"
format_stamp: "Formatted at 2026-06-20 19:09:16 on dist-test-slave-6wrf"
server_key: "febc819fd99ffbdae6165c76569693a3"
server_key_iv: "4860b1cfc162548b6f04777187e70574"
server_key_version: "encryptionkey@0"
I20260620 19:09:16.511785 20797 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:09:16.514479 20812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:16.515710 20797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:16.515869 20797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "c3f29237f61e4691b254f27720d56f05"
format_stamp: "Formatted at 2026-06-20 19:09:16 on dist-test-slave-6wrf"
server_key: "febc819fd99ffbdae6165c76569693a3"
server_key_iv: "4860b1cfc162548b6f04777187e70574"
server_key_version: "encryptionkey@0"
I20260620 19:09:16.515986 20797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:16.527163 20797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:16.531263 20797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:16.531564 20797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:16.539911 20797 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:40229
I20260620 19:09:16.539974 20864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:40229 every 8 connection(s)
I20260620 19:09:16.540913 20797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:16.543617 20865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:16.544109 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 20797
I20260620 19:09:16.544286 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:16.544606 1642 external_mini_cluster.cc:1428] Setting key d496abb5f3b5d1f0cc3c765c7cbcb989
I20260620 19:09:16.550859 20865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05: Bootstrap starting.
I20260620 19:09:16.553301 20865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05: Neither blocks nor log segments found. Creating new log.
Jun 20 19:09:16 dist-test-slave-6wrf krb5kdc[20784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:16.554085 20865 log.cc:824] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:16.555944 20865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05: No bootstrap required, opened a new log
I20260620 19:09:16.559158 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:16.546047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39764 (local address 127.1.154.190:40229)
0620 19:09:16.546469 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:16.546487 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:16.546529 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:09:16.547271 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:16.547293 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:16.547710 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:16.548059 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:16.548771 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:16.550274 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:16.551189 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:16.551822 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:16.552027 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:16.554646 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:16.554651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:16.554664 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:16.554691 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:16.556440 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:16.556875 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:16.556879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:16.556883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:16.556988 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:16.557276 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:16.557281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:16.557282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:16.557720 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:16.557915 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:16.558194 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 19:09:16.558470 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":131,"threads_started":1}
I20260620 19:09:16.559537 20865 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f29237f61e4691b254f27720d56f05" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40229 } }
I20260620 19:09:16.559775 20865 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:16.559859 20865 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f29237f61e4691b254f27720d56f05, State: Initialized, Role: FOLLOWER
I20260620 19:09:16.560300 20865 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f29237f61e4691b254f27720d56f05" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40229 } }
I20260620 19:09:16.560456 20865 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:16.560541 20865 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:16.560655 20865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:16.561618 20865 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f29237f61e4691b254f27720d56f05" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40229 } }
I20260620 19:09:16.561934 20865 leader_election.cc:302] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f29237f61e4691b254f27720d56f05; no voters:
I20260620 19:09:16.562240 20865 leader_election.cc:288] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:16.562480 20870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:16.562719 20870 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [term 1 LEADER]: Becoming Leader. State: Replica: c3f29237f61e4691b254f27720d56f05, State: Running, Role: LEADER
I20260620 19:09:16.563020 20870 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f29237f61e4691b254f27720d56f05" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40229 } }
I20260620 19:09:16.563416 20865 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:16.564596 20871 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3f29237f61e4691b254f27720d56f05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f29237f61e4691b254f27720d56f05" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40229 } } }
I20260620 19:09:16.564715 20871 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:16.565023 20872 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f29237f61e4691b254f27720d56f05. Latest consensus state: current_term: 1 leader_uuid: "c3f29237f61e4691b254f27720d56f05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f29237f61e4691b254f27720d56f05" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 40229 } } }
I20260620 19:09:16.565122 20872 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05 [sys.catalog]: This master's current role is: LEADER
W20260620 19:09:16.568130 20885 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:09:16.568215 20885 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:09:16.568295 20879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:16.568971 20879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:16.573917 20879 catalog_manager.cc:1347] Generated new cluster ID: a2a10931596f49fd8b981ea98da1e9b9
I20260620 19:09:16.574003 20879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:16.586306 20879 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:16.587637 20879 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:16.593948 20879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05: Generated new TSK 0
I20260620 19:09:16.594738 20879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.154.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
Entry for principal kudu/127.1.154.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab.
I20260620 19:09:16.635231 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40229
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:16.750644 20891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:16.751132 20891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:16.751243 20891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:16.757236 20891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:16.757393 20891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:16.757515 20891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:09:16.763866 20891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.129.keytab
--principal=kudu/127.1.154.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:16.765163 20891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:16.766283 20891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:16.776510 20896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:16.776593 20891 server_base.cc:1034] running on GCE node
W20260620 19:09:16.776510 20899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:16.776527 20897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:16.777226 20891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:16.777949 20891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:16.779142 20891 hybrid_clock.cc:648] HybridClock initialized: now 1781982556779111 us; error 43 us; skew 500 ppm
Jun 20 19:09:16 dist-test-slave-6wrf krb5kdc[20784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982556, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:16.783142 20891 init.cc:377] Logged in from keytab as kudu/127.1.154.129@KRBTEST.COM (short username kudu)
I20260620 19:09:16.784269 20891 webserver.cc:460] Webserver started at http://127.1.154.129:40535/ using document root <none> and password file <none>
I20260620 19:09:16.784868 20891 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:16.784962 20891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:16.785182 20891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:16.787658 20891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "71a6918700ed49458e9bef6ae0865113"
format_stamp: "Formatted at 2026-06-20 19:09:16 on dist-test-slave-6wrf"
server_key: "ac8a832d21eefd20d6718cbb4072a126"
server_key_iv: "bfa3a1faccd77c0102fdfc07023fc97d"
server_key_version: "encryptionkey@0"
I20260620 19:09:16.788230 20891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "71a6918700ed49458e9bef6ae0865113"
format_stamp: "Formatted at 2026-06-20 19:09:16 on dist-test-slave-6wrf"
server_key: "ac8a832d21eefd20d6718cbb4072a126"
server_key_iv: "bfa3a1faccd77c0102fdfc07023fc97d"
server_key_version: "encryptionkey@0"
I20260620 19:09:16.792343 20891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260620 19:09:16.795364 20906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:16.797024 20891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:09:16.797137 20891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "71a6918700ed49458e9bef6ae0865113"
format_stamp: "Formatted at 2026-06-20 19:09:16 on dist-test-slave-6wrf"
server_key: "ac8a832d21eefd20d6718cbb4072a126"
server_key_iv: "bfa3a1faccd77c0102fdfc07023fc97d"
server_key_version: "encryptionkey@0"
I20260620 19:09:16.797271 20891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:16.815316 20891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:16.819748 20891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:16.819919 20891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:16.820529 20891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:16.821377 20891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:16.821426 20891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:16.821471 20891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:16.821486 20891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:16.831401 20891 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:35075
I20260620 19:09:16.831431 21019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:35075 every 8 connection(s)
I20260620 19:09:16.832293 20891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:09:16.834031 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 20891
I20260620 19:09:16.834175 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:09:16.834504 1642 external_mini_cluster.cc:1428] Setting key 86a0a9070bc4d70afc5ba6916a588b0c
Jun 20 19:09:16 dist-test-slave-6wrf krb5kdc[20784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982556, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.129@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.154.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.130@KRBTEST.COM" created.
I20260620 19:09:16.849380 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:16.834274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:59689 (local address 127.1.154.190:40229)
0620 19:09:16.834419 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:16.834423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:16.835368 (+ 945us) server_negotiation.cc:407] Connection header received
0620 19:09:16.837170 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:16.837175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:16.837272 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:16.837402 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:16.838790 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:16.839851 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:16.841180 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:16.841590 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:16.841743 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:16.844651 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:16.844662 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:16.844669 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:16.844725 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:16.846943 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:16.847640 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:16.847643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:16.847644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:16.847759 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:16.848093 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:16.848096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:16.848097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:16.848304 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:16.848409 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:16.848993 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 19:09:16.849158 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:09:16.850374 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:16.834693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:40229 (local address 127.1.154.129:59689)
0620 19:09:16.835204 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:16.835260 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:16.836851 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:16.837632 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:16.837645 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:16.838105 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:16.838593 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:16.838605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:16.840007 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:16.840010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:16.841022 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:16.841038 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:16.841766 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:16.841769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:16.841927 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:16.842607 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:16.842625 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:16.844309 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:16.847082 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:16.847087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:16.847103 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:16.847518 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:16.847855 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:16.847857 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:16.847859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:16.847998 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:16.848430 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:16.848435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:16.848702 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 19:09:16.848893 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 19:09:16.849210 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":151,"threads_started":1}
I20260620 19:09:16.852047 21020 heartbeater.cc:347] Connected to a master server at 127.1.154.190:40229
I20260620 19:09:16.852303 21020 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:16.852883 21020 heartbeater.cc:511] Master 127.1.154.190:40229 requested a full tablet report, sending...
I20260620 19:09:16.854561 20829 ts_manager.cc:194] Registered new tserver with Master: 71a6918700ed49458e9bef6ae0865113 (127.1.154.129:35075)
I20260620 19:09:16.855659 20829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.129@KRBTEST.COM'} at 127.1.154.129:59689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
Entry for principal kudu/127.1.154.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab.
I20260620 19:09:16.868534 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40229
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:16.979053 21025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:16.979357 21025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:16.979430 21025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:16.983501 21025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:16.983593 21025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:16.983685 21025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:09:16.989457 21025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.130.keytab
--principal=kudu/127.1.154.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:16.990586 21025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:16.991617 21025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:17.003170 21033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:17.003229 21030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:17.003193 21031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:17.003424 21025 server_base.cc:1034] running on GCE node
I20260620 19:09:17.004097 21025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:17.004789 21025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:17.005992 21025 hybrid_clock.cc:648] HybridClock initialized: now 1781982557005970 us; error 50 us; skew 500 ppm
Jun 20 19:09:17 dist-test-slave-6wrf krb5kdc[20784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982557, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:17.009335 21025 init.cc:377] Logged in from keytab as kudu/127.1.154.130@KRBTEST.COM (short username kudu)
I20260620 19:09:17.010653 21025 webserver.cc:460] Webserver started at http://127.1.154.130:40605/ using document root <none> and password file <none>
I20260620 19:09:17.011266 21025 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:17.011337 21025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:17.011555 21025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:17.013650 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "e2f13c79f39c49ef887edf84f1bd4d79"
format_stamp: "Formatted at 2026-06-20 19:09:17 on dist-test-slave-6wrf"
server_key: "c21d3011754192cd2a11676ba3dec003"
server_key_iv: "82caf3bad827fba60ebcf239fb981532"
server_key_version: "encryptionkey@0"
I20260620 19:09:17.014254 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "e2f13c79f39c49ef887edf84f1bd4d79"
format_stamp: "Formatted at 2026-06-20 19:09:17 on dist-test-slave-6wrf"
server_key: "c21d3011754192cd2a11676ba3dec003"
server_key_iv: "82caf3bad827fba60ebcf239fb981532"
server_key_version: "encryptionkey@0"
I20260620 19:09:17.019114 21025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 19:09:17.021672 21040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:17.023025 21025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:09:17.023157 21025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "e2f13c79f39c49ef887edf84f1bd4d79"
format_stamp: "Formatted at 2026-06-20 19:09:17 on dist-test-slave-6wrf"
server_key: "c21d3011754192cd2a11676ba3dec003"
server_key_iv: "82caf3bad827fba60ebcf239fb981532"
server_key_version: "encryptionkey@0"
I20260620 19:09:17.023272 21025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:17.041572 21025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:17.045106 21025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:17.045336 21025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:17.045993 21025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:17.046823 21025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:17.046873 21025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:17.046937 21025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:17.046969 21025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:17.056456 21025 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:45861
I20260620 19:09:17.056514 21153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:45861 every 8 connection(s)
I20260620 19:09:17.057405 21025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:09:17.066308 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 21025
I20260620 19:09:17.066468 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:09:17.066742 1642 external_mini_cluster.cc:1428] Setting key e8371a3b5f6bb8e7003b4d4189f4ea29
Jun 20 19:09:17 dist-test-slave-6wrf krb5kdc[20784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982557, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.130@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:17.077301 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.060455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:34559 (local address 127.1.154.190:40229)
0620 19:09:17.060643 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:17.060648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:17.061916 (+ 1268us) server_negotiation.cc:407] Connection header received
0620 19:09:17.063398 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:17.063402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:17.063463 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:17.063569 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:17.064901 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.065562 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.067015 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.067286 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.067411 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.071988 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:17.071996 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:17.072004 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:17.072154 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:17.074521 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:17.075104 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:17.075110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:17.075114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:17.075216 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:17.075559 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:17.075565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:17.075568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:17.075836 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:17.075965 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:17.076927 (+ 962us) server_negotiation.cc:299] Negotiation successful
0620 19:09:17.077072 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
WARNING: no policy specified for kudu/127.1.154.131@KRBTEST.COM; defaulting to no policy
I20260620 19:09:17.078161 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.061011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:40229 (local address 127.1.154.130:34559)
0620 19:09:17.061734 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:17.061784 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:17.063116 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:17.063746 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:17.063753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:17.064196 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:17.064758 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:17.064771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.065731 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.065737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:17.066890 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:17.066901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.067455 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.067459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:17.067655 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.069881 (+ 2226us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:17.069897 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:17.071704 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:17.074661 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:17.074666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:17.074678 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:17.074995 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:17.075328 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:17.075330 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:17.075332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:17.075463 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:17.076079 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:17.076088 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:17.076621 (+ 533us) client_negotiation.cc:769] Sending connection context
0620 19:09:17.076841 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 19:09:17.077126 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":209,"threads_started":1}
I20260620 19:09:17.080159 21154 heartbeater.cc:347] Connected to a master server at 127.1.154.190:40229
I20260620 19:09:17.080524 21154 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.131@KRBTEST.COM" created.
I20260620 19:09:17.081205 21154 heartbeater.cc:511] Master 127.1.154.190:40229 requested a full tablet report, sending...
I20260620 19:09:17.082718 20829 ts_manager.cc:194] Registered new tserver with Master: e2f13c79f39c49ef887edf84f1bd4d79 (127.1.154.130:45861)
I20260620 19:09:17.083756 20829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.130@KRBTEST.COM'} at 127.1.154.130:34559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
Entry for principal kudu/127.1.154.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab.
I20260620 19:09:17.099557 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40229
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:17.226621 21159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:17.227101 21159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:17.227196 21159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:17.231600 21159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:17.231673 21159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:17.231770 21159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:09:17.237226 21159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.131.keytab
--principal=kudu/127.1.154.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:40229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:17.238418 21159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:17.239452 21159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:17.249322 21167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:17.249368 21164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:17.249343 21165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:17.249436 21159 server_base.cc:1034] running on GCE node
I20260620 19:09:17.250051 21159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:17.250737 21159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:17.251926 21159 hybrid_clock.cc:648] HybridClock initialized: now 1781982557251912 us; error 28 us; skew 500 ppm
Jun 20 19:09:17 dist-test-slave-6wrf krb5kdc[20784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982557, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:17.254840 21159 init.cc:377] Logged in from keytab as kudu/127.1.154.131@KRBTEST.COM (short username kudu)
I20260620 19:09:17.255822 21159 webserver.cc:460] Webserver started at http://127.1.154.131:32907/ using document root <none> and password file <none>
I20260620 19:09:17.256637 21159 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:17.256712 21159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:17.256956 21159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:17.259133 21159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "59f756d8ddd546ff8b5d281d6fe87d4b"
format_stamp: "Formatted at 2026-06-20 19:09:17 on dist-test-slave-6wrf"
server_key: "ce039631f7674646e4ec076538635aa1"
server_key_iv: "c626441b09db545572a81a98e4b21444"
server_key_version: "encryptionkey@0"
I20260620 19:09:17.259694 21159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "59f756d8ddd546ff8b5d281d6fe87d4b"
format_stamp: "Formatted at 2026-06-20 19:09:17 on dist-test-slave-6wrf"
server_key: "ce039631f7674646e4ec076538635aa1"
server_key_iv: "c626441b09db545572a81a98e4b21444"
server_key_version: "encryptionkey@0"
I20260620 19:09:17.264133 21159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:09:17.267208 21174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:17.268573 21159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:09:17.268692 21159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "59f756d8ddd546ff8b5d281d6fe87d4b"
format_stamp: "Formatted at 2026-06-20 19:09:17 on dist-test-slave-6wrf"
server_key: "ce039631f7674646e4ec076538635aa1"
server_key_iv: "c626441b09db545572a81a98e4b21444"
server_key_version: "encryptionkey@0"
I20260620 19:09:17.268798 21159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:17.305721 21159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:17.309106 21159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:17.309264 21159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:17.310014 21159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:17.310879 21159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:17.310961 21159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:17.311031 21159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:17.311080 21159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:17.321422 21159 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:36805
I20260620 19:09:17.321449 21287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:36805 every 8 connection(s)
I20260620 19:09:17.322431 21159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:09:17.328795 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 21159
I20260620 19:09:17.328965 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:09:17.329267 1642 external_mini_cluster.cc:1428] Setting key e429bc1bdd4d6c6ccec62d4f1249708b
Jun 20 19:09:17 dist-test-slave-6wrf krb5kdc[20784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982557, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.131@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:17.337682 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.324425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:33859 (local address 127.1.154.190:40229)
0620 19:09:17.324618 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:17.324622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:17.325424 (+ 802us) server_negotiation.cc:407] Connection header received
0620 19:09:17.326817 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:17.326820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:17.326869 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:17.326944 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:17.328210 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.328795 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.330220 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.330445 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.330647 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.333227 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:17.333230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:17.333233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:17.333283 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:17.334803 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:17.335825 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:17.335829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:17.335831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:17.335947 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:17.336503 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:17.336507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:17.336509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:17.336712 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:17.336851 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:17.337348 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 19:09:17.337476 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 19:09:17.338546 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.324795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:40229 (local address 127.1.154.131:33859)
0620 19:09:17.325254 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:17.325304 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:17.326591 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:17.327112 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:17.327118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:17.327496 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:17.328048 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:17.328063 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.328949 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.328953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:17.330097 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:17.330106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.330688 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.330692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:17.330902 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.331564 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:17.331590 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:17.333060 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:17.335141 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:17.335150 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:17.335175 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:17.335681 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:17.336162 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:17.336166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:17.336168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:17.336354 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:17.336832 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:17.336838 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:17.337101 (+ 263us) client_negotiation.cc:769] Sending connection context
0620 19:09:17.337304 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 19:09:17.337696 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":123,"threads_started":1}
I20260620 19:09:17.339973 21288 heartbeater.cc:347] Connected to a master server at 127.1.154.190:40229
I20260620 19:09:17.340234 21288 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:17.340746 21288 heartbeater.cc:511] Master 127.1.154.190:40229 requested a full tablet report, sending...
I20260620 19:09:17.342343 20829 ts_manager.cc:194] Registered new tserver with Master: 59f756d8ddd546ff8b5d281d6fe87d4b (127.1.154.131:36805)
I20260620 19:09:17.343144 20829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.154.131@KRBTEST.COM'} at 127.1.154.131:33859
I20260620 19:09:17.344344 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:09:17.354255 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.346359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.1.154.190:40229)
0620 19:09:17.346518 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:17.346523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:17.346665 (+ 142us) server_negotiation.cc:407] Connection header received
0620 19:09:17.346808 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:17.346811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:17.346876 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:17.346962 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:17.347673 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.348186 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.349037 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.349273 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.349386 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.350840 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:17.350844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:17.350847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:17.350872 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:17.352518 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:17.352952 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:17.352956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:17.352958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:17.353040 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:17.353321 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:17.353325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:17.353326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:17.353547 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:17.353729 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:17.353971 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 19:09:17.354083 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 19:09:17.357324 20829 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:09:17.360311 20829 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:09:17.376662 21303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.371345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.1.154.129:35075)
0620 19:09:17.371674 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:17.371682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:17.371902 (+ 220us) server_negotiation.cc:407] Connection header received
0620 19:09:17.372009 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:17.372014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:17.372176 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:17.372662 (+ 486us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:17.372780 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.374052 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.375462 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.376008 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.376134 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.376283 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:17.376425 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 19:09:17.376537 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":138,"threads_started":1}
I20260620 19:09:17.376647 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.371184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:35075 (local address 127.0.0.1:45484)
0620 19:09:17.371812 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:17.371824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:17.371910 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:17.372375 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:17.372378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:17.372397 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:17.372515 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:17.372521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.374181 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.374184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:17.375368 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:17.375378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.376134 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.376138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:17.376238 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.376273 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 19:09:17.376381 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 19:09:17.376496 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":37,"thread_start_us":91,"threads_started":1}
I20260620 19:09:17.376960 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.370500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:45861 (local address 127.0.0.1:37794)
0620 19:09:17.371049 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:17.371098 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:17.371245 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:17.372208 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:17.372215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:17.372240 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:17.372426 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:17.372435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.373876 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.373880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:17.375281 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:17.375291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.376455 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.376458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:17.376621 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.376662 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 19:09:17.376732 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:09:17.376825 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":40,"spinlock_wait_cycles":7040,"thread_start_us":87,"threads_started":1}
I20260620 19:09:17.377058 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.371594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.1.154.131:36805)
0620 19:09:17.372194 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:17.372205 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:17.372224 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:09:17.372288 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:17.372292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:17.372453 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:17.372588 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:17.372805 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.373745 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.375269 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.376128 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.376247 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.376382 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:17.376817 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 19:09:17.376912 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":97,"threads_started":1}
I20260620 19:09:17.377424 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.370686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.1.154.130:45861)
0620 19:09:17.371737 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:17.371749 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:17.371772 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:09:17.371846 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:17.371852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:17.372043 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:17.372214 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:17.372552 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.373743 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.375414 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.376200 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.376287 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.376407 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:17.377182 (+ 775us) server_negotiation.cc:299] Negotiation successful
0620 19:09:17.377278 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":937,"thread_start_us":114,"threads_started":1}
I20260620 19:09:17.377668 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.370692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:36805 (local address 127.0.0.1:37752)
0620 19:09:17.371051 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:17.371099 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:17.371241 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:17.372578 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:17.372581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:17.372597 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:17.372685 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:17.372694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.373905 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.373908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:17.375144 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:17.375153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.376448 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.376451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:17.376618 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.376666 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 19:09:17.376732 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:09:17.377393 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":685,"spinlock_wait_cycles":5760,"thread_start_us":110,"threads_started":1}
I20260620 19:09:17.379918 21222 tablet_service.cc:1467] Processing CreateTablet for tablet e13f6a5878ca437997e42f1fdff11b69 (DEFAULT_TABLE table=test-table [id=e38108aaac4948159b8a3a713a3b51c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:17.379951 21088 tablet_service.cc:1467] Processing CreateTablet for tablet e13f6a5878ca437997e42f1fdff11b69 (DEFAULT_TABLE table=test-table [id=e38108aaac4948159b8a3a713a3b51c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:17.379951 20953 tablet_service.cc:1467] Processing CreateTablet for tablet e13f6a5878ca437997e42f1fdff11b69 (DEFAULT_TABLE table=test-table [id=e38108aaac4948159b8a3a713a3b51c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:17.380903 21222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13f6a5878ca437997e42f1fdff11b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:17.380914 20953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13f6a5878ca437997e42f1fdff11b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:17.380914 21088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13f6a5878ca437997e42f1fdff11b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:17.387308 21305 tablet_bootstrap.cc:492] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b: Bootstrap starting.
I20260620 19:09:17.388051 21306 tablet_bootstrap.cc:492] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113: Bootstrap starting.
I20260620 19:09:17.389026 21307 tablet_bootstrap.cc:492] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79: Bootstrap starting.
I20260620 19:09:17.389724 21305 tablet_bootstrap.cc:654] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:17.389850 21306 tablet_bootstrap.cc:654] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:17.390666 21306 log.cc:824] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:17.390916 21305 log.cc:824] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:17.391428 21307 tablet_bootstrap.cc:654] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:17.392195 21307 log.cc:824] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:17.393172 21305 tablet_bootstrap.cc:492] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b: No bootstrap required, opened a new log
I20260620 19:09:17.393369 21305 ts_tablet_manager.cc:1397] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260620 19:09:17.394044 21307 tablet_bootstrap.cc:492] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79: No bootstrap required, opened a new log
I20260620 19:09:17.394192 21306 tablet_bootstrap.cc:492] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113: No bootstrap required, opened a new log
I20260620 19:09:17.394207 21307 ts_tablet_manager.cc:1397] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:09:17.394405 21306 ts_tablet_manager.cc:1397] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:09:17.397349 21305 raft_consensus.cc:348] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f13c79f39c49ef887edf84f1bd4d79" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45861 } } peers { permanent_uuid: "59f756d8ddd546ff8b5d281d6fe87d4b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36805 } } peers { permanent_uuid: "71a6918700ed49458e9bef6ae0865113" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 35075 } }
I20260620 19:09:17.397658 21305 raft_consensus.cc:374] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:17.397720 21305 raft_consensus.cc:729] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f756d8ddd546ff8b5d281d6fe87d4b, State: Initialized, Role: FOLLOWER
I20260620 19:09:17.397794 21306 raft_consensus.cc:348] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f13c79f39c49ef887edf84f1bd4d79" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45861 } } peers { permanent_uuid: "59f756d8ddd546ff8b5d281d6fe87d4b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36805 } } peers { permanent_uuid: "71a6918700ed49458e9bef6ae0865113" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 35075 } }
I20260620 19:09:17.398068 21306 raft_consensus.cc:374] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:17.398125 21306 raft_consensus.cc:729] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a6918700ed49458e9bef6ae0865113, State: Initialized, Role: FOLLOWER
I20260620 19:09:17.397830 21307 raft_consensus.cc:348] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f13c79f39c49ef887edf84f1bd4d79" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45861 } } peers { permanent_uuid: "59f756d8ddd546ff8b5d281d6fe87d4b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36805 } } peers { permanent_uuid: "71a6918700ed49458e9bef6ae0865113" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 35075 } }
I20260620 19:09:17.398263 21307 raft_consensus.cc:374] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:17.398190 21305 consensus_queue.cc:260] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f13c79f39c49ef887edf84f1bd4d79" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45861 } } peers { permanent_uuid: "59f756d8ddd546ff8b5d281d6fe87d4b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36805 } } peers { permanent_uuid: "71a6918700ed49458e9bef6ae0865113" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 35075 } }
I20260620 19:09:17.398387 21307 raft_consensus.cc:729] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f13c79f39c49ef887edf84f1bd4d79, State: Initialized, Role: FOLLOWER
I20260620 19:09:17.398555 21306 consensus_queue.cc:260] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f13c79f39c49ef887edf84f1bd4d79" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45861 } } peers { permanent_uuid: "59f756d8ddd546ff8b5d281d6fe87d4b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36805 } } peers { permanent_uuid: "71a6918700ed49458e9bef6ae0865113" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 35075 } }
I20260620 19:09:17.398789 21307 consensus_queue.cc:260] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f13c79f39c49ef887edf84f1bd4d79" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45861 } } peers { permanent_uuid: "59f756d8ddd546ff8b5d281d6fe87d4b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36805 } } peers { permanent_uuid: "71a6918700ed49458e9bef6ae0865113" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 35075 } }
I20260620 19:09:17.399314 21020 heartbeater.cc:503] Master 127.1.154.190:40229 was elected leader, sending a full tablet report...
I20260620 19:09:17.399592 21307 ts_tablet_manager.cc:1428] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:09:17.399746 21306 ts_tablet_manager.cc:1428] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:09:17.400358 21154 heartbeater.cc:503] Master 127.1.154.190:40229 was elected leader, sending a full tablet report...
I20260620 19:09:17.401189 21305 ts_tablet_manager.cc:1428] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260620 19:09:17.401501 21288 heartbeater.cc:503] Master 127.1.154.190:40229 was elected leader, sending a full tablet report...
I20260620 19:09:17.458492 21311 raft_consensus.cc:482] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:09:17.458730 21311 raft_consensus.cc:504] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f13c79f39c49ef887edf84f1bd4d79" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45861 } } peers { permanent_uuid: "59f756d8ddd546ff8b5d281d6fe87d4b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36805 } } peers { permanent_uuid: "71a6918700ed49458e9bef6ae0865113" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 35075 } }
I20260620 19:09:17.459980 21311 leader_election.cc:288] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2f13c79f39c49ef887edf84f1bd4d79 (127.1.154.130:45861), 71a6918700ed49458e9bef6ae0865113 (127.1.154.129:35075)
I20260620 19:09:17.463824 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.460277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:35075 (local address 127.1.154.131:60985)
0620 19:09:17.460517 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:17.460558 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:17.460669 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:17.460943 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:17.460946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:17.460961 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:17.461086 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:17.461091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.461903 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.461909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:17.462682 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:17.462690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.463392 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.463396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:17.463504 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.463532 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 19:09:17.463620 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 19:09:17.463688 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 19:09:17.463824 21303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.460419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:60985 (local address 127.1.154.129:35075)
0620 19:09:17.460557 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:17.460564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:17.460634 (+ 70us) server_negotiation.cc:407] Connection header received
0620 19:09:17.460795 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:17.460805 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:17.460847 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:17.460933 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:17.461193 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.461789 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.462811 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.463277 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.463401 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.463456 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:17.463636 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 19:09:17.463704 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:09:17.464021 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.460284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:45861 (local address 127.1.154.131:50053)
0620 19:09:17.460738 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:17.460749 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:17.460876 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:17.461139 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:17.461142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:17.461157 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:17.461282 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:17.461287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.462013 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.462017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:17.462784 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:17.462792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.463585 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.463588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:17.463697 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.463724 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:09:17.463815 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 19:09:17.463882 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":126,"thread_start_us":130,"threads_started":1}
I20260620 19:09:17.464013 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.460360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:50053 (local address 127.1.154.130:45861)
0620 19:09:17.460577 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:17.460583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:17.460839 (+ 256us) server_negotiation.cc:407] Connection header received
0620 19:09:17.460990 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:17.460993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:17.461043 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:17.461130 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:17.461386 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.461890 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.462904 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.463468 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.463609 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.463660 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:17.463831 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 19:09:17.463893 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:09:17.464690 20974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13f6a5878ca437997e42f1fdff11b69" candidate_uuid: "59f756d8ddd546ff8b5d281d6fe87d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a6918700ed49458e9bef6ae0865113" is_pre_election: true
I20260620 19:09:17.464692 21108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13f6a5878ca437997e42f1fdff11b69" candidate_uuid: "59f756d8ddd546ff8b5d281d6fe87d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f13c79f39c49ef887edf84f1bd4d79" is_pre_election: true
I20260620 19:09:17.464987 20974 raft_consensus.cc:2454] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f756d8ddd546ff8b5d281d6fe87d4b in term 0.
I20260620 19:09:17.464989 21108 raft_consensus.cc:2454] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f756d8ddd546ff8b5d281d6fe87d4b in term 0.
I20260620 19:09:17.465503 21176 leader_election.cc:302] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f756d8ddd546ff8b5d281d6fe87d4b, e2f13c79f39c49ef887edf84f1bd4d79; no voters:
I20260620 19:09:17.465814 21311 raft_consensus.cc:2781] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:09:17.465901 21311 raft_consensus.cc:482] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:09:17.465955 21311 raft_consensus.cc:3037] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:17.466992 21311 raft_consensus.cc:504] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f13c79f39c49ef887edf84f1bd4d79" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45861 } } peers { permanent_uuid: "59f756d8ddd546ff8b5d281d6fe87d4b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36805 } } peers { permanent_uuid: "71a6918700ed49458e9bef6ae0865113" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 35075 } }
I20260620 19:09:17.467312 21311 leader_election.cc:288] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [CANDIDATE]: Term 1 election: Requested vote from peers e2f13c79f39c49ef887edf84f1bd4d79 (127.1.154.130:45861), 71a6918700ed49458e9bef6ae0865113 (127.1.154.129:35075)
I20260620 19:09:17.467643 21108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13f6a5878ca437997e42f1fdff11b69" candidate_uuid: "59f756d8ddd546ff8b5d281d6fe87d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f13c79f39c49ef887edf84f1bd4d79"
I20260620 19:09:17.467662 20974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13f6a5878ca437997e42f1fdff11b69" candidate_uuid: "59f756d8ddd546ff8b5d281d6fe87d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a6918700ed49458e9bef6ae0865113"
I20260620 19:09:17.467769 20974 raft_consensus.cc:3037] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:17.467769 21108 raft_consensus.cc:3037] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:17.469007 20974 raft_consensus.cc:2454] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f756d8ddd546ff8b5d281d6fe87d4b in term 1.
I20260620 19:09:17.469007 21108 raft_consensus.cc:2454] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f756d8ddd546ff8b5d281d6fe87d4b in term 1.
I20260620 19:09:17.469305 21177 leader_election.cc:302] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f756d8ddd546ff8b5d281d6fe87d4b, 71a6918700ed49458e9bef6ae0865113; no voters:
I20260620 19:09:17.469506 21311 raft_consensus.cc:2781] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:17.469846 21311 raft_consensus.cc:686] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [term 1 LEADER]: Becoming Leader. State: Replica: 59f756d8ddd546ff8b5d281d6fe87d4b, State: Running, Role: LEADER
I20260620 19:09:17.470196 21311 consensus_queue.cc:237] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f13c79f39c49ef887edf84f1bd4d79" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45861 } } peers { permanent_uuid: "59f756d8ddd546ff8b5d281d6fe87d4b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36805 } } peers { permanent_uuid: "71a6918700ed49458e9bef6ae0865113" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 35075 } }
I20260620 19:09:17.473867 20828 catalog_manager.cc:5495] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f756d8ddd546ff8b5d281d6fe87d4b (127.1.154.131). New cstate: current_term: 1 leader_uuid: "59f756d8ddd546ff8b5d281d6fe87d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f13c79f39c49ef887edf84f1bd4d79" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f756d8ddd546ff8b5d281d6fe87d4b" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 36805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a6918700ed49458e9bef6ae0865113" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 35075 } health_report { overall_health: UNKNOWN } } }
I20260620 19:09:17.501416 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.497505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.1.154.131:36805)
0620 19:09:17.497705 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:17.497708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:17.497722 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:09:17.497794 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:17.497796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:17.497848 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:17.497932 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:17.498177 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.498674 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.499852 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.500092 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.500181 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.500513 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:17.501005 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:17.501095 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:17.501193 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 19:09:17.501244 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 19:09:17.507153 21108 raft_consensus.cc:1261] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79 [term 1 FOLLOWER]: Refusing update from remote peer 59f756d8ddd546ff8b5d281d6fe87d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:17.507153 20974 raft_consensus.cc:1261] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113 [term 1 FOLLOWER]: Refusing update from remote peer 59f756d8ddd546ff8b5d281d6fe87d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:17.507853 21311 consensus_queue.cc:1035] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f13c79f39c49ef887edf84f1bd4d79" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 45861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:17.508051 21315 consensus_queue.cc:1035] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a6918700ed49458e9bef6ae0865113" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:17.518139 21319 mvcc.cc:205] Tried to move back new op lower bound from 7299000555541643264 to 7299000555400572928. Current Snapshot: MvccSnapshot[applied={T|T < 7299000555541643264}]
I20260620 19:09:17.525068 20829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:09:17.525235 20829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:09:17.529047 20829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3f29237f61e4691b254f27720d56f05: Sending DeleteTablet for 3 replicas of tablet e13f6a5878ca437997e42f1fdff11b69
I20260620 19:09:17.529945 21222 tablet_service.cc:1514] Processing DeleteTablet for tablet e13f6a5878ca437997e42f1fdff11b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:17 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:37752
I20260620 19:09:17.529947 21088 tablet_service.cc:1514] Processing DeleteTablet for tablet e13f6a5878ca437997e42f1fdff11b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:17 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:37794
I20260620 19:09:17.529947 20953 tablet_service.cc:1514] Processing DeleteTablet for tablet e13f6a5878ca437997e42f1fdff11b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:17 UTC) from {username='kudu', principal='kudu/127.1.154.190@KRBTEST.COM'} at 127.0.0.1:45484
I20260620 19:09:17.530422 21329 tablet_replica.cc:330] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113: stopping tablet replica
I20260620 19:09:17.530624 21328 tablet_replica.cc:330] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79: stopping tablet replica
I20260620 19:09:17.530745 21329 raft_consensus.cc:2229] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:09:17.531021 21328 raft_consensus.cc:2229] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:09:17.531085 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 20891
I20260620 19:09:17.531085 21329 raft_consensus.cc:2258] T e13f6a5878ca437997e42f1fdff11b69 P 71a6918700ed49458e9bef6ae0865113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:17.531725 21328 raft_consensus.cc:2258] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:17.531786 21330 tablet_replica.cc:330] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b: stopping tablet replica
I20260620 19:09:17.532073 21330 raft_consensus.cc:2229] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:09:17.532452 21330 raft_consensus.cc:2258] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:17.533749 21328 ts_tablet_manager.cc:1905] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:09:17.534127 21330 ts_tablet_manager.cc:1905] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:09:17.536790 21328 ts_tablet_manager.cc:1918] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:09:17.536963 21328 log.cc:1196] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/e13f6a5878ca437997e42f1fdff11b69
I20260620 19:09:17.537207 21330 ts_tablet_manager.cc:1918] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:09:17.537303 21330 log.cc:1196] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/e13f6a5878ca437997e42f1fdff11b69
I20260620 19:09:17.537472 21328 ts_tablet_manager.cc:1939] T e13f6a5878ca437997e42f1fdff11b69 P e2f13c79f39c49ef887edf84f1bd4d79: Deleting consensus metadata
I20260620 19:09:17.537652 21330 ts_tablet_manager.cc:1939] T e13f6a5878ca437997e42f1fdff11b69 P 59f756d8ddd546ff8b5d281d6fe87d4b: Deleting consensus metadata
I20260620 19:09:17.538758 20815 catalog_manager.cc:4841] TS 59f756d8ddd546ff8b5d281d6fe87d4b (127.1.154.131:36805): tablet e13f6a5878ca437997e42f1fdff11b69 (table test-table [id=e38108aaac4948159b8a3a713a3b51c5]) successfully deleted
I20260620 19:09:17.538760 20814 catalog_manager.cc:4841] TS e2f13c79f39c49ef887edf84f1bd4d79 (127.1.154.130:45861): tablet e13f6a5878ca437997e42f1fdff11b69 (table test-table [id=e38108aaac4948159b8a3a713a3b51c5]) successfully deleted
W20260620 19:09:17.541401 20815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:35075 (error 108)
I20260620 19:09:17.541507 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 21025
I20260620 19:09:17.544292 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.544003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:35075 (local address 127.0.0.1:45500)
0620 19:09:17.544165 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:17.544222 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:35075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260620 19:09:17.544430 20815 catalog_manager.cc:4584] TS 71a6918700ed49458e9bef6ae0865113 (127.1.154.129:35075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e13f6a5878ca437997e42f1fdff11b69: Network error: Client connection negotiation failed: client connection to 127.1.154.129:35075: connect: Connection refused (error 111)
I20260620 19:09:17.550343 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 21159
I20260620 19:09:17.556311 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 20797
2026-06-20T19:09:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2394 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T19:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:17Z Disabled control of system clock
I20260620 19:09:17.587749 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:46283
--webserver_interface=127.1.154.190
--webserver_port=0
--builtin_ntp_servers=127.1.154.148:40761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:46283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:09:17.706032 21335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:17.706321 21335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:17.706362 21335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:17.710779 21335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:17.710841 21335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:17.710857 21335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:17.710876 21335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:17.710893 21335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:17.716173 21335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:46283
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:46283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:17.717259 21335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:17.718699 21335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:17.727828 21343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:17.727828 21340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:17.728130 21341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:17.728261 21335 server_base.cc:1034] running on GCE node
I20260620 19:09:17.728950 21335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:17.730131 21335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:17.731335 21335 hybrid_clock.cc:648] HybridClock initialized: now 1781982557731305 us; error 44 us; skew 500 ppm
I20260620 19:09:17.733227 21335 webserver.cc:460] Webserver started at http://127.1.154.190:43873/ using document root <none> and password file <none>
I20260620 19:09:17.733894 21335 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:17.733984 21335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:17.734216 21335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:17.736238 21335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "c3d235bb3f1b4d58a15f6d8692366b5a"
format_stamp: "Formatted at 2026-06-20 19:09:17 on dist-test-slave-6wrf"
server_key: "9110c81de6471b152937cc962ca3092c"
server_key_iv: "61d00afb0f019f3508eba197e1ae3092"
server_key_version: "encryptionkey@0"
I20260620 19:09:17.736805 21335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "c3d235bb3f1b4d58a15f6d8692366b5a"
format_stamp: "Formatted at 2026-06-20 19:09:17 on dist-test-slave-6wrf"
server_key: "9110c81de6471b152937cc962ca3092c"
server_key_iv: "61d00afb0f019f3508eba197e1ae3092"
server_key_version: "encryptionkey@0"
I20260620 19:09:17.740507 21335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 19:09:17.743285 21349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:17.744562 21335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:09:17.744725 21335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "c3d235bb3f1b4d58a15f6d8692366b5a"
format_stamp: "Formatted at 2026-06-20 19:09:17 on dist-test-slave-6wrf"
server_key: "9110c81de6471b152937cc962ca3092c"
server_key_iv: "61d00afb0f019f3508eba197e1ae3092"
server_key_version: "encryptionkey@0"
I20260620 19:09:17.744843 21335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:17.770277 21335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:17.771004 21335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:17.771222 21335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:17.778697 21335 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:46283
I20260620 19:09:17.779009 21401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:46283 every 8 connection(s)
I20260620 19:09:17.779753 21335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:17.783206 21402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:17.785035 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 21335
I20260620 19:09:17.785257 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:17.785912 1642 external_mini_cluster.cc:1428] Setting key bb3ae237cc6d313f031de6bc06892306
I20260620 19:09:17.789767 21402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a: Bootstrap starting.
I20260620 19:09:17.792874 21402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:17.793831 21402 log.cc:824] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:17.795900 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:17.787465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.1.154.190:46283)
0620 19:09:17.788081 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:17.788095 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:17.788131 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:09:17.788740 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:17.788755 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:17.789131 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:17.789401 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:17.789815 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.790807 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.792170 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:17.792618 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:17.792740 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:17.793851 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:17.793856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:17.793868 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:17.794425 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:17.794495 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:17.794523 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:17.794636 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:17.794866 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 19:09:17.795178 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":131,"threads_started":1}
I20260620 19:09:17.795994 21402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a: No bootstrap required, opened a new log
I20260620 19:09:17.799702 21402 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d235bb3f1b4d58a15f6d8692366b5a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46283 } }
I20260620 19:09:17.799937 21402 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:17.799983 21402 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d235bb3f1b4d58a15f6d8692366b5a, State: Initialized, Role: FOLLOWER
I20260620 19:09:17.800462 21402 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d235bb3f1b4d58a15f6d8692366b5a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46283 } }
I20260620 19:09:17.800591 21402 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:17.800635 21402 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:17.800700 21402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:17.801685 21402 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d235bb3f1b4d58a15f6d8692366b5a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46283 } }
I20260620 19:09:17.801982 21402 leader_election.cc:302] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3d235bb3f1b4d58a15f6d8692366b5a; no voters:
I20260620 19:09:17.802317 21402 leader_election.cc:288] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:17.802464 21407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:17.802726 21407 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [term 1 LEADER]: Becoming Leader. State: Replica: c3d235bb3f1b4d58a15f6d8692366b5a, State: Running, Role: LEADER
I20260620 19:09:17.803032 21407 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d235bb3f1b4d58a15f6d8692366b5a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46283 } }
I20260620 19:09:17.803721 21402 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:17.804584 21408 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3d235bb3f1b4d58a15f6d8692366b5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d235bb3f1b4d58a15f6d8692366b5a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46283 } } }
I20260620 19:09:17.804720 21408 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:17.805114 21416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:17.805048 21409 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3d235bb3f1b4d58a15f6d8692366b5a. Latest consensus state: current_term: 1 leader_uuid: "c3d235bb3f1b4d58a15f6d8692366b5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d235bb3f1b4d58a15f6d8692366b5a" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 46283 } } }
I20260620 19:09:17.805255 21409 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:17.808537 21416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:17.814304 21416 catalog_manager.cc:1347] Generated new cluster ID: 13d225fb4c1c494faf39d862b5492c45
I20260620 19:09:17.814486 21416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:17.836058 21416 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:17.837262 21416 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:17.847301 21416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a: Generated new TSK 0
I20260620 19:09:17.848160 21416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:17.857412 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.129:0
--local_ip_for_outbound_sockets=127.1.154.129
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46283
--builtin_ntp_servers=127.1.154.148:40761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:09:17.968955 21426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:17.969229 21426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:17.969272 21426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:17.973515 21426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:17.973578 21426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:17.973655 21426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.129
I20260620 19:09:17.978307 21426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.154.129
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.129
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:17.979405 21426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:17.980770 21426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:17.991058 21431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:17.991062 21434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:17.991210 21426 server_base.cc:1034] running on GCE node
W20260620 19:09:17.991101 21432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:17.991798 21426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:17.992540 21426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:17.993779 21426 hybrid_clock.cc:648] HybridClock initialized: now 1781982557993743 us; error 50 us; skew 500 ppm
I20260620 19:09:17.995744 21426 webserver.cc:460] Webserver started at http://127.1.154.129:46155/ using document root <none> and password file <none>
I20260620 19:09:17.996336 21426 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:17.996430 21426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:17.996675 21426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:17.998821 21426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/data/instance:
uuid: "ff0383c03ffa4f8fb6ca05d10769da1e"
format_stamp: "Formatted at 2026-06-20 19:09:17 on dist-test-slave-6wrf"
server_key: "0dd026f3ff4be5a5fdde35cb40b83f24"
server_key_iv: "86ff4fb9aec4ac4ad4f741a9e327d8b8"
server_key_version: "encryptionkey@0"
I20260620 19:09:17.999429 21426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance:
uuid: "ff0383c03ffa4f8fb6ca05d10769da1e"
format_stamp: "Formatted at 2026-06-20 19:09:17 on dist-test-slave-6wrf"
server_key: "0dd026f3ff4be5a5fdde35cb40b83f24"
server_key_iv: "86ff4fb9aec4ac4ad4f741a9e327d8b8"
server_key_version: "encryptionkey@0"
I20260620 19:09:18.003671 21426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:09:18.006556 21440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:18.007854 21426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:18.007965 21426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/wal
uuid: "ff0383c03ffa4f8fb6ca05d10769da1e"
format_stamp: "Formatted at 2026-06-20 19:09:17 on dist-test-slave-6wrf"
server_key: "0dd026f3ff4be5a5fdde35cb40b83f24"
server_key_iv: "86ff4fb9aec4ac4ad4f741a9e327d8b8"
server_key_version: "encryptionkey@0"
I20260620 19:09:18.008064 21426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:18.028620 21426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:18.029349 21426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:18.029516 21426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:18.030241 21426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:18.031313 21426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:18.031369 21426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:18.031414 21426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:18.031481 21426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:18.042109 21426 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.129:41881
I20260620 19:09:18.042243 21553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.129:41881 every 8 connection(s)
I20260620 19:09:18.043215 21426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/data/info.pb
I20260620 19:09:18.044274 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 21426
I20260620 19:09:18.044427 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-0/wal/instance
I20260620 19:09:18.044759 1642 external_mini_cluster.cc:1428] Setting key 27fa0cd9d561cf8fd7f41fe16a92150e
I20260620 19:09:18.047493 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.130:0
--local_ip_for_outbound_sockets=127.1.154.130
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46283
--builtin_ntp_servers=127.1.154.148:40761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:09:18.059010 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.046196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:46701 (local address 127.1.154.190:46283)
0620 19:09:18.046344 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:18.046349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:18.048351 (+ 2002us) server_negotiation.cc:407] Connection header received
0620 19:09:18.050925 (+ 2574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:18.050930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:18.051010 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:18.051125 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:18.052488 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.053740 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.055607 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.056044 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.056195 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.058121 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:18.058128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:18.058135 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:18.058163 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:18.058193 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:18.058213 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:18.058313 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:18.058689 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 19:09:18.058776 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:09:18.060045 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.046839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:46283 (local address 127.1.154.129:46701)
0620 19:09:18.048103 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:18.048202 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:18.050540 (+ 2338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:18.051335 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:18.051344 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:18.051488 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:18.052261 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.052280 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.054014 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.054019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.055391 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.055409 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.056294 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.056297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.056497 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.057470 (+ 973us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:09:18.057472 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:18.057532 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:09:18.057898 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:18.058385 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:18.058396 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:18.058403 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 19:09:18.058597 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 19:09:18.059127 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":186,"threads_started":1}
I20260620 19:09:18.061496 21554 heartbeater.cc:347] Connected to a master server at 127.1.154.190:46283
I20260620 19:09:18.061812 21554 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:18.062363 21554 heartbeater.cc:511] Master 127.1.154.190:46283 requested a full tablet report, sending...
I20260620 19:09:18.063988 21366 ts_manager.cc:194] Registered new tserver with Master: ff0383c03ffa4f8fb6ca05d10769da1e (127.1.154.129:41881)
I20260620 19:09:18.065169 21366 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.154.129:46701
W20260620 19:09:18.161161 21557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:18.161420 21557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:18.161458 21557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:18.165591 21557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:18.165659 21557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:18.165714 21557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.130
I20260620 19:09:18.171720 21557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.154.130
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.130
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:18.172851 21557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:18.174249 21557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:18.183964 21562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:18.183945 21563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:18.183945 21565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:18.184365 21557 server_base.cc:1034] running on GCE node
I20260620 19:09:18.184762 21557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:18.185400 21557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:18.186652 21557 hybrid_clock.cc:648] HybridClock initialized: now 1781982558186600 us; error 50 us; skew 500 ppm
I20260620 19:09:18.189352 21557 webserver.cc:460] Webserver started at http://127.1.154.130:44683/ using document root <none> and password file <none>
I20260620 19:09:18.189998 21557 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:18.190068 21557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:18.190281 21557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:18.192341 21557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/data/instance:
uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c"
format_stamp: "Formatted at 2026-06-20 19:09:18 on dist-test-slave-6wrf"
server_key: "d4a70228338807cad61bcf9b2b39d7e4"
server_key_iv: "e4d65ad4ef5d5033246d99b82e2e609e"
server_key_version: "encryptionkey@0"
I20260620 19:09:18.192885 21557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance:
uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c"
format_stamp: "Formatted at 2026-06-20 19:09:18 on dist-test-slave-6wrf"
server_key: "d4a70228338807cad61bcf9b2b39d7e4"
server_key_iv: "e4d65ad4ef5d5033246d99b82e2e609e"
server_key_version: "encryptionkey@0"
I20260620 19:09:18.196549 21557 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:09:18.199468 21571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:18.201352 21557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:09:18.201591 21557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/wal
uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c"
format_stamp: "Formatted at 2026-06-20 19:09:18 on dist-test-slave-6wrf"
server_key: "d4a70228338807cad61bcf9b2b39d7e4"
server_key_iv: "e4d65ad4ef5d5033246d99b82e2e609e"
server_key_version: "encryptionkey@0"
I20260620 19:09:18.201814 21557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:18.219841 21557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:18.220664 21557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:18.220865 21557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:18.221464 21557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:18.222492 21557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:18.222565 21557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:18.222630 21557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:18.222681 21557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:18.233186 21557 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.130:38325
I20260620 19:09:18.233202 21684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.130:38325 every 8 connection(s)
I20260620 19:09:18.234138 21557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/data/info.pb
I20260620 19:09:18.236748 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 21557
I20260620 19:09:18.236932 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/wal/instance
I20260620 19:09:18.237262 1642 external_mini_cluster.cc:1428] Setting key fe8d280219a22de0fc31e5b10113fdce
I20260620 19:09:18.239289 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.154.131:0
--local_ip_for_outbound_sockets=127.1.154.131
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46283
--builtin_ntp_servers=127.1.154.148:40761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:09:18.243717 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.236031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.130:50547 (local address 127.1.154.190:46283)
0620 19:09:18.236242 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:18.236247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:18.237251 (+ 1004us) server_negotiation.cc:407] Connection header received
0620 19:09:18.238944 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:18.238947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:18.238997 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:18.239099 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:18.240028 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.240566 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.241713 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.242005 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.242122 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.243228 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:18.243238 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:18.243240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:18.243251 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:18.243268 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:18.243276 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:18.243355 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:18.243514 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 19:09:18.243572 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 19:09:18.244527 21687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.236394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:46283 (local address 127.1.154.130:50547)
0620 19:09:18.237094 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:18.237143 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:18.238730 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:18.239271 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:18.239277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:18.239361 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:18.239896 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.239909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.240691 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.240695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.241547 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.241555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.242167 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.242171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.242337 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.242948 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:09:18.242950 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:18.242987 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:09:18.243046 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:18.243356 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:18.243367 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:18.243375 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:09:18.243502 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 19:09:18.243762 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":141,"threads_started":1}
I20260620 19:09:18.245724 21685 heartbeater.cc:347] Connected to a master server at 127.1.154.190:46283
I20260620 19:09:18.245978 21685 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:18.246949 21685 heartbeater.cc:511] Master 127.1.154.190:46283 requested a full tablet report, sending...
I20260620 19:09:18.248538 21366 ts_manager.cc:194] Registered new tserver with Master: e84c5e0fc47c44b7a1602c8fe6f86d5c (127.1.154.130:38325)
I20260620 19:09:18.249200 21366 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.154.130:50547
W20260620 19:09:18.357105 21688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:18.357403 21688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:18.357470 21688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:18.361538 21688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:18.361658 21688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:18.361768 21688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.154.131
I20260620 19:09:18.366636 21688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.154.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.154.131
--webserver_port=0
--tserver_master_addrs=127.1.154.190:46283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.154.131
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:18.367758 21688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:18.369125 21688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:18.379503 21696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:18.379539 21693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:18.379558 21694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:18.379858 21688 server_base.cc:1034] running on GCE node
I20260620 19:09:18.380820 21688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:18.381497 21688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:18.382736 21688 hybrid_clock.cc:648] HybridClock initialized: now 1781982558382722 us; error 44 us; skew 500 ppm
I20260620 19:09:18.387425 21688 webserver.cc:460] Webserver started at http://127.1.154.131:43203/ using document root <none> and password file <none>
I20260620 19:09:18.388062 21688 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:18.388161 21688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:18.388401 21688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:18.390539 21688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/data/instance:
uuid: "e7cd4c6092fd4577bbdcb0df849d6795"
format_stamp: "Formatted at 2026-06-20 19:09:18 on dist-test-slave-6wrf"
server_key: "4723bf94ee620879b502579058450605"
server_key_iv: "22b428591b6188e962c996de8f00325a"
server_key_version: "encryptionkey@0"
I20260620 19:09:18.391110 21688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance:
uuid: "e7cd4c6092fd4577bbdcb0df849d6795"
format_stamp: "Formatted at 2026-06-20 19:09:18 on dist-test-slave-6wrf"
server_key: "4723bf94ee620879b502579058450605"
server_key_iv: "22b428591b6188e962c996de8f00325a"
server_key_version: "encryptionkey@0"
I20260620 19:09:18.394711 21688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 19:09:18.397780 21702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:18.399084 21688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 19:09:18.399245 21688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/wal
uuid: "e7cd4c6092fd4577bbdcb0df849d6795"
format_stamp: "Formatted at 2026-06-20 19:09:18 on dist-test-slave-6wrf"
server_key: "4723bf94ee620879b502579058450605"
server_key_iv: "22b428591b6188e962c996de8f00325a"
server_key_version: "encryptionkey@0"
I20260620 19:09:18.399370 21688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:18.420073 21688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:18.420814 21688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:18.421005 21688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:18.421584 21688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:18.422605 21688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:18.422680 21688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:18.422750 21688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:18.422806 21688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:18.432276 21688 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.131:41531
I20260620 19:09:18.432505 21815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.131:41531 every 8 connection(s)
I20260620 19:09:18.433295 21688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/data/info.pb
I20260620 19:09:18.436210 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 21688
I20260620 19:09:18.436319 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/wal/instance
I20260620 19:09:18.436650 1642 external_mini_cluster.cc:1428] Setting key 6d0995bec44822539f287dba726f2c2f
I20260620 19:09:18.443593 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.435218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.131:34821 (local address 127.1.154.190:46283)
0620 19:09:18.435385 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:18.435390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:18.436604 (+ 1214us) server_negotiation.cc:407] Connection header received
0620 19:09:18.438541 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:18.438544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:18.438633 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:18.438726 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:18.439753 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.440474 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.441760 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.441976 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.442066 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.443194 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:18.443197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:18.443199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:18.443211 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:18.443228 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:18.443235 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:18.443296 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:18.443436 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 19:09:18.443483 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 19:09:18.444415 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.435605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:46283 (local address 127.1.154.131:34821)
0620 19:09:18.436381 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:18.436434 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:18.438271 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:18.438873 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:18.438879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:18.438961 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:18.439564 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.439578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.440634 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.440637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.441634 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.441645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.442147 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.442150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.442311 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.442919 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:09:18.442921 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:18.442967 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:09:18.443032 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:18.443311 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:18.443317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:18.443321 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:09:18.443414 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 19:09:18.443615 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260620 19:09:18.445662 21816 heartbeater.cc:347] Connected to a master server at 127.1.154.190:46283
I20260620 19:09:18.445889 21816 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:18.446398 21816 heartbeater.cc:511] Master 127.1.154.190:46283 requested a full tablet report, sending...
I20260620 19:09:18.447860 21366 ts_manager.cc:194] Registered new tserver with Master: e7cd4c6092fd4577bbdcb0df849d6795 (127.1.154.131:41531)
I20260620 19:09:18.448413 21366 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.154.131:34821
I20260620 19:09:18.451094 1642 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:09:18.458294 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.453089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.1.154.190:46283)
0620 19:09:18.453276 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:18.453280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:18.453883 (+ 603us) server_negotiation.cc:407] Connection header received
0620 19:09:18.453998 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:18.454001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:18.454042 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:18.454117 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:18.454525 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.455178 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.456099 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.456349 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.456445 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.457420 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:18.457425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:18.457427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:18.457446 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:18.457481 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:18.457491 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:18.457640 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:18.458112 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 19:09:18.458178 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260620 19:09:18.462852 21366 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:09:18.467646 21366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:09:18.483317 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.477632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60892 (local address 127.1.154.129:41881)
0620 19:09:18.478388 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:18.478399 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:18.478421 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:09:18.478494 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:18.478498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:18.478666 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:18.478803 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:18.479233 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.480350 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.481781 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.482419 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.482538 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.482734 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:18.483009 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 19:09:18.483131 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":103,"threads_started":1}
I20260620 19:09:18.483588 21828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.476844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:38325 (local address 127.0.0.1:39106)
0620 19:09:18.477663 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:18.477714 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:18.477851 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:18.478633 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:18.478639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:18.478661 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:18.478804 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.478811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.479944 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.479950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.481418 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.481429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.483068 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.483071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.483157 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.483179 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:09:18.483229 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:09:18.483315 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":1,"spinlock_wait_cycles":896,"thread_start_us":84,"threads_started":1}
I20260620 19:09:18.483958 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.477226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:41881 (local address 127.0.0.1:60892)
0620 19:09:18.478058 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:18.478070 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:18.478135 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:18.478953 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:18.478956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:18.478973 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:18.479081 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.479108 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.480467 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.480470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.481657 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.481666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.482573 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.482578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.482740 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.482797 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 19:09:18.482959 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 19:09:18.483142 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260620 19:09:18.483640 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.477024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:41531 (local address 127.0.0.1:37282)
0620 19:09:18.477992 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:18.478007 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:18.478135 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:18.478906 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:18.478911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:18.478931 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:18.479031 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.479037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.480744 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.480748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.481882 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.482059 (+ 177us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.483174 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.483179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.483284 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.483320 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 19:09:18.483379 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 19:09:18.483504 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":219,"spinlock_wait_cycles":11008,"thread_start_us":118,"threads_started":1}
I20260620 19:09:18.484251 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.478155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.1.154.131:41531)
0620 19:09:18.478520 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:18.478532 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:18.478551 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:09:18.478614 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:18.478617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:18.478770 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:18.478903 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:18.479151 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.480616 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.482277 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.483004 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.483748 (+ 744us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.483923 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:18.483974 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 19:09:18.484086 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":151,"threads_started":1}
I20260620 19:09:18.485126 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.477635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.1.154.130:38325)
0620 19:09:18.478127 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:18.478138 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:18.478162 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:09:18.478238 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:18.478242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:18.478455 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:18.478643 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:18.478964 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.479824 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.481780 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.482911 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.484571 (+ 1660us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.484781 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:18.484838 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 19:09:18.484952 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":133,"threads_started":1}
I20260620 19:09:18.487310 21619 tablet_service.cc:1467] Processing CreateTablet for tablet 9e70b6cbe92945aeb615d3fa56c6a66b (DEFAULT_TABLE table=test-table [id=1febc6ce470b496f8bef1f790888058c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:18.487195 21750 tablet_service.cc:1467] Processing CreateTablet for tablet 9e70b6cbe92945aeb615d3fa56c6a66b (DEFAULT_TABLE table=test-table [id=1febc6ce470b496f8bef1f790888058c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:18.487211 21487 tablet_service.cc:1467] Processing CreateTablet for tablet 9e70b6cbe92945aeb615d3fa56c6a66b (DEFAULT_TABLE table=test-table [id=1febc6ce470b496f8bef1f790888058c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:18.488260 21619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e70b6cbe92945aeb615d3fa56c6a66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:18.488266 21487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e70b6cbe92945aeb615d3fa56c6a66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:18.488266 21750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e70b6cbe92945aeb615d3fa56c6a66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:18.496522 21833 tablet_bootstrap.cc:492] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c: Bootstrap starting.
I20260620 19:09:18.497017 21835 tablet_bootstrap.cc:492] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795: Bootstrap starting.
I20260620 19:09:18.498579 21833 tablet_bootstrap.cc:654] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:18.498740 21835 tablet_bootstrap.cc:654] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:18.499388 21835 log.cc:824] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:18.499388 21833 log.cc:824] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:18.500077 21834 tablet_bootstrap.cc:492] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e: Bootstrap starting.
I20260620 19:09:18.502049 21833 tablet_bootstrap.cc:492] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c: No bootstrap required, opened a new log
I20260620 19:09:18.502089 21834 tablet_bootstrap.cc:654] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:18.502271 21833 ts_tablet_manager.cc:1397] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:09:18.502775 21835 tablet_bootstrap.cc:492] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795: No bootstrap required, opened a new log
I20260620 19:09:18.502907 21834 log.cc:824] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:18.502997 21835 ts_tablet_manager.cc:1397] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:09:18.504942 21834 tablet_bootstrap.cc:492] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e: No bootstrap required, opened a new log
I20260620 19:09:18.505162 21834 ts_tablet_manager.cc:1397] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:09:18.506569 21833 raft_consensus.cc:348] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38325 } } peers { permanent_uuid: "e7cd4c6092fd4577bbdcb0df849d6795" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 41531 } } peers { permanent_uuid: "ff0383c03ffa4f8fb6ca05d10769da1e" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 41881 } }
I20260620 19:09:18.506866 21833 raft_consensus.cc:374] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:18.506947 21833 raft_consensus.cc:729] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e84c5e0fc47c44b7a1602c8fe6f86d5c, State: Initialized, Role: FOLLOWER
I20260620 19:09:18.507364 21835 raft_consensus.cc:348] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38325 } } peers { permanent_uuid: "e7cd4c6092fd4577bbdcb0df849d6795" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 41531 } } peers { permanent_uuid: "ff0383c03ffa4f8fb6ca05d10769da1e" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 41881 } }
I20260620 19:09:18.507432 21833 consensus_queue.cc:260] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38325 } } peers { permanent_uuid: "e7cd4c6092fd4577bbdcb0df849d6795" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 41531 } } peers { permanent_uuid: "ff0383c03ffa4f8fb6ca05d10769da1e" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 41881 } }
I20260620 19:09:18.507720 21835 raft_consensus.cc:374] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:18.507812 21835 raft_consensus.cc:729] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7cd4c6092fd4577bbdcb0df849d6795, State: Initialized, Role: FOLLOWER
I20260620 19:09:18.508260 21835 consensus_queue.cc:260] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38325 } } peers { permanent_uuid: "e7cd4c6092fd4577bbdcb0df849d6795" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 41531 } } peers { permanent_uuid: "ff0383c03ffa4f8fb6ca05d10769da1e" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 41881 } }
I20260620 19:09:18.508502 21833 ts_tablet_manager.cc:1428] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:09:18.508915 21685 heartbeater.cc:503] Master 127.1.154.190:46283 was elected leader, sending a full tablet report...
I20260620 19:09:18.508925 21816 heartbeater.cc:503] Master 127.1.154.190:46283 was elected leader, sending a full tablet report...
I20260620 19:09:18.509106 21835 ts_tablet_manager.cc:1428] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:09:18.510264 21834 raft_consensus.cc:348] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38325 } } peers { permanent_uuid: "e7cd4c6092fd4577bbdcb0df849d6795" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 41531 } } peers { permanent_uuid: "ff0383c03ffa4f8fb6ca05d10769da1e" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 41881 } }
I20260620 19:09:18.510505 21834 raft_consensus.cc:374] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:18.510605 21834 raft_consensus.cc:729] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff0383c03ffa4f8fb6ca05d10769da1e, State: Initialized, Role: FOLLOWER
I20260620 19:09:18.511016 21834 consensus_queue.cc:260] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38325 } } peers { permanent_uuid: "e7cd4c6092fd4577bbdcb0df849d6795" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 41531 } } peers { permanent_uuid: "ff0383c03ffa4f8fb6ca05d10769da1e" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 41881 } }
I20260620 19:09:18.511606 21554 heartbeater.cc:503] Master 127.1.154.190:46283 was elected leader, sending a full tablet report...
I20260620 19:09:18.511853 21834 ts_tablet_manager.cc:1428] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 19:09:18.534466 21841 raft_consensus.cc:482] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:09:18.534684 21841 raft_consensus.cc:504] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38325 } } peers { permanent_uuid: "e7cd4c6092fd4577bbdcb0df849d6795" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 41531 } } peers { permanent_uuid: "ff0383c03ffa4f8fb6ca05d10769da1e" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 41881 } }
I20260620 19:09:18.535880 21841 leader_election.cc:288] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e84c5e0fc47c44b7a1602c8fe6f86d5c (127.1.154.130:38325), e7cd4c6092fd4577bbdcb0df849d6795 (127.1.154.131:41531)
I20260620 19:09:18.540943 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.536059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.130:38325 (local address 127.1.154.129:48473)
0620 19:09:18.536426 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:18.536447 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:18.536567 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:18.536859 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:18.536862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:18.536876 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:18.537025 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.537032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.538078 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.538083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.539292 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.539347 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.540398 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.540401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.540536 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.540564 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 19:09:18.540680 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 19:09:18.540774 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260620 19:09:18.540978 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.536187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:48473 (local address 127.1.154.130:38325)
0620 19:09:18.536336 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:18.536341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:18.536552 (+ 211us) server_negotiation.cc:407] Connection header received
0620 19:09:18.536693 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:18.536697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:18.536749 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:18.536836 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:18.537270 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.537950 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.539537 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.540248 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.540482 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.540566 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:18.540691 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 19:09:18.540790 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:09:18.541569 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.536451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.154.129:56597 (local address 127.1.154.131:41531)
0620 19:09:18.536581 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:18.536585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:18.536942 (+ 357us) server_negotiation.cc:407] Connection header received
0620 19:09:18.537041 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:18.537044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:18.537100 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:18.537185 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:18.537724 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.538570 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.540135 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.540801 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.540919 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.540966 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:18.541367 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 19:09:18.541449 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:09:18.541703 21639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e70b6cbe92945aeb615d3fa56c6a66b" candidate_uuid: "ff0383c03ffa4f8fb6ca05d10769da1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c" is_pre_election: true
I20260620 19:09:18.541981 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.536467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.131:41531 (local address 127.1.154.129:56597)
0620 19:09:18.536836 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:18.536851 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:18.536946 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:18.537306 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:18.537309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:18.537326 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:18.537440 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.537446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.538769 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.538772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.539968 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:18.539981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.541061 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.541067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:18.541216 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.541244 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 19:09:18.541769 (+ 525us) client_negotiation.cc:240] Negotiation successful
0620 19:09:18.541857 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":142,"threads_started":1}
I20260620 19:09:18.542030 21639 raft_consensus.cc:2454] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff0383c03ffa4f8fb6ca05d10769da1e in term 0.
I20260620 19:09:18.542539 21442 leader_election.cc:302] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e84c5e0fc47c44b7a1602c8fe6f86d5c, ff0383c03ffa4f8fb6ca05d10769da1e; no voters:
I20260620 19:09:18.542544 21770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e70b6cbe92945aeb615d3fa56c6a66b" candidate_uuid: "ff0383c03ffa4f8fb6ca05d10769da1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7cd4c6092fd4577bbdcb0df849d6795" is_pre_election: true
I20260620 19:09:18.542820 21770 raft_consensus.cc:2454] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff0383c03ffa4f8fb6ca05d10769da1e in term 0.
I20260620 19:09:18.542820 21841 raft_consensus.cc:2781] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:09:18.542909 21841 raft_consensus.cc:482] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:09:18.542953 21841 raft_consensus.cc:3037] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:18.544080 21841 raft_consensus.cc:504] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38325 } } peers { permanent_uuid: "e7cd4c6092fd4577bbdcb0df849d6795" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 41531 } } peers { permanent_uuid: "ff0383c03ffa4f8fb6ca05d10769da1e" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 41881 } }
I20260620 19:09:18.544401 21841 leader_election.cc:288] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [CANDIDATE]: Term 1 election: Requested vote from peers e84c5e0fc47c44b7a1602c8fe6f86d5c (127.1.154.130:38325), e7cd4c6092fd4577bbdcb0df849d6795 (127.1.154.131:41531)
I20260620 19:09:18.544765 21770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e70b6cbe92945aeb615d3fa56c6a66b" candidate_uuid: "ff0383c03ffa4f8fb6ca05d10769da1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7cd4c6092fd4577bbdcb0df849d6795"
I20260620 19:09:18.544783 21639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e70b6cbe92945aeb615d3fa56c6a66b" candidate_uuid: "ff0383c03ffa4f8fb6ca05d10769da1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c"
W20260620 19:09:18.544860 21555 tablet.cc:2369] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:09:18.544898 21770 raft_consensus.cc:3037] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:18.544907 21639 raft_consensus.cc:3037] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:18.545985 21770 raft_consensus.cc:2454] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff0383c03ffa4f8fb6ca05d10769da1e in term 1.
I20260620 19:09:18.546142 21639 raft_consensus.cc:2454] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff0383c03ffa4f8fb6ca05d10769da1e in term 1.
I20260620 19:09:18.546293 21441 leader_election.cc:302] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7cd4c6092fd4577bbdcb0df849d6795, ff0383c03ffa4f8fb6ca05d10769da1e; no voters:
I20260620 19:09:18.546474 21841 raft_consensus.cc:2781] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:18.546702 21841 raft_consensus.cc:686] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [term 1 LEADER]: Becoming Leader. State: Replica: ff0383c03ffa4f8fb6ca05d10769da1e, State: Running, Role: LEADER
I20260620 19:09:18.547036 21841 consensus_queue.cc:237] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38325 } } peers { permanent_uuid: "e7cd4c6092fd4577bbdcb0df849d6795" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 41531 } } peers { permanent_uuid: "ff0383c03ffa4f8fb6ca05d10769da1e" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 41881 } }
I20260620 19:09:18.550184 21365 catalog_manager.cc:5495] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e reported cstate change: term changed from 0 to 1, leader changed from <none> to ff0383c03ffa4f8fb6ca05d10769da1e (127.1.154.129). New cstate: current_term: 1 leader_uuid: "ff0383c03ffa4f8fb6ca05d10769da1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7cd4c6092fd4577bbdcb0df849d6795" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff0383c03ffa4f8fb6ca05d10769da1e" member_type: VOTER last_known_addr { host: "127.1.154.129" port: 41881 } health_report { overall_health: HEALTHY } } }
I20260620 19:09:18.565758 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.562130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60906 (local address 127.1.154.129:41881)
0620 19:09:18.562253 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:18.562257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:18.562271 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:09:18.562436 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:18.562438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:18.562494 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:18.562626 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:18.562866 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.563377 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.564270 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:18.564489 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:18.564600 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:18.564910 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:18.565333 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:18.565470 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:18.565519 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 19:09:18.565576 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:09:18.572124 21639 raft_consensus.cc:1261] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c [term 1 FOLLOWER]: Refusing update from remote peer ff0383c03ffa4f8fb6ca05d10769da1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:18.572575 21770 raft_consensus.cc:1261] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795 [term 1 FOLLOWER]: Refusing update from remote peer ff0383c03ffa4f8fb6ca05d10769da1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:18.572817 21843 consensus_queue.cc:1035] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e84c5e0fc47c44b7a1602c8fe6f86d5c" member_type: VOTER last_known_addr { host: "127.1.154.130" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:18.573544 21843 consensus_queue.cc:1035] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7cd4c6092fd4577bbdcb0df849d6795" member_type: VOTER last_known_addr { host: "127.1.154.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:18.581468 21847 mvcc.cc:205] Tried to move back new op lower bound from 7299000559903498240 to 7299000559811092480. Current Snapshot: MvccSnapshot[applied={T|T < 7299000559903498240}]
I20260620 19:09:18.585007 21846 mvcc.cc:205] Tried to move back new op lower bound from 7299000559903498240 to 7299000559811092480. Current Snapshot: MvccSnapshot[applied={T|T < 7299000559903498240 or (T in {7299000559903498240})}]
I20260620 19:09:18.585114 21848 mvcc.cc:205] Tried to move back new op lower bound from 7299000559903498240 to 7299000559811092480. Current Snapshot: MvccSnapshot[applied={T|T < 7299000559903498240}]
I20260620 19:09:18.590862 21366 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:09:18.591081 21366 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:09:18.593554 21366 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3d235bb3f1b4d58a15f6d8692366b5a: Sending DeleteTablet for 3 replicas of tablet 9e70b6cbe92945aeb615d3fa56c6a66b
I20260620 19:09:18.594981 21750 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e70b6cbe92945aeb615d3fa56c6a66b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:18 UTC) from {username='slave'} at 127.0.0.1:37282
I20260620 19:09:18.594973 21619 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e70b6cbe92945aeb615d3fa56c6a66b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:18 UTC) from {username='slave'} at 127.0.0.1:39106
I20260620 19:09:18.595788 21855 tablet_replica.cc:330] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795: stopping tablet replica
I20260620 19:09:18.595820 21487 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e70b6cbe92945aeb615d3fa56c6a66b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:18 UTC) from {username='slave'} at 127.0.0.1:60892
I20260620 19:09:18.596165 21855 raft_consensus.cc:2229] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:09:18.596315 21857 tablet_replica.cc:330] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e: stopping tablet replica
I20260620 19:09:18.596498 21855 raft_consensus.cc:2258] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:18.596647 21857 raft_consensus.cc:2229] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:09:18.597188 21857 raft_consensus.cc:2258] T 9e70b6cbe92945aeb615d3fa56c6a66b P ff0383c03ffa4f8fb6ca05d10769da1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:18.597368 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 21426
I20260620 19:09:18.598145 21856 tablet_replica.cc:330] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c: stopping tablet replica
I20260620 19:09:18.598337 21855 ts_tablet_manager.cc:1905] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:09:18.598429 21856 raft_consensus.cc:2229] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:09:18.598696 21856 raft_consensus.cc:2258] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:18.600250 21856 ts_tablet_manager.cc:1905] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:09:18.601999 21855 ts_tablet_manager.cc:1918] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:09:18.602108 21855 log.cc:1196] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-2/wal/wals/9e70b6cbe92945aeb615d3fa56c6a66b
I20260620 19:09:18.602533 21855 ts_tablet_manager.cc:1939] T 9e70b6cbe92945aeb615d3fa56c6a66b P e7cd4c6092fd4577bbdcb0df849d6795: Deleting consensus metadata
I20260620 19:09:18.602648 21856 ts_tablet_manager.cc:1918] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:09:18.602741 21856 log.cc:1196] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c: Deleting WAL directory at /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982293404112-1642-0/minicluster-data/ts-1/wal/wals/9e70b6cbe92945aeb615d3fa56c6a66b
I20260620 19:09:18.603037 21856 ts_tablet_manager.cc:1939] T 9e70b6cbe92945aeb615d3fa56c6a66b P e84c5e0fc47c44b7a1602c8fe6f86d5c: Deleting consensus metadata
I20260620 19:09:18.604022 21350 catalog_manager.cc:4841] TS e7cd4c6092fd4577bbdcb0df849d6795 (127.1.154.131:41531): tablet 9e70b6cbe92945aeb615d3fa56c6a66b (table test-table [id=1febc6ce470b496f8bef1f790888058c]) successfully deleted
I20260620 19:09:18.604183 21351 catalog_manager.cc:4841] TS e84c5e0fc47c44b7a1602c8fe6f86d5c (127.1.154.130:38325): tablet 9e70b6cbe92945aeb615d3fa56c6a66b (table test-table [id=1febc6ce470b496f8bef1f790888058c]) successfully deleted
I20260620 19:09:18.606590 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 21557
W20260620 19:09:18.606606 21353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.154.129:41881 (error 108)
I20260620 19:09:18.608413 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:18.608045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.129:41881 (local address 127.0.0.1:60916)
0620 19:09:18.608265 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:18.608337 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.129:41881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260620 19:09:18.608605 21353 catalog_manager.cc:4584] TS ff0383c03ffa4f8fb6ca05d10769da1e (127.1.154.129:41881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e70b6cbe92945aeb615d3fa56c6a66b: Network error: Client connection negotiation failed: client connection to 127.1.154.129:41881: connect: Connection refused (error 111)
I20260620 19:09:18.614210 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 21688
I20260620 19:09:18.620424 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 21335
2026-06-20T19:09:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1063 ms)
[----------] 6 tests from EncryptionPolicyTest (10454 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:18 dist-test-slave-6wrf krb5kdc[21861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:18 dist-test-slave-6wrf krb5kdc[21861](info): set up 2 sockets
Jun 20 19:09:18 dist-test-slave-6wrf krb5kdc[21861](info): commencing operation
krb5kdc: starting...
W20260620 19:09:19.701795 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:19 dist-test-slave-6wrf krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:09:19.835290 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40717
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:19.946645 21874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:19.946985 21874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:19.947070 21874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:19.951355 21874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:19.951476 21874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:19.951517 21874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:19.951571 21874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:19.951608 21874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:19.957039 21874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40717
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:19.958176 21874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:19.959189 21874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:19.968290 21880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:19.968384 21882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:19.968315 21879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:19.968639 21874 server_base.cc:1034] running on GCE node
I20260620 19:09:19.969179 21874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:19.970430 21874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:19.971630 21874 hybrid_clock.cc:648] HybridClock initialized: now 1781982559971598 us; error 46 us; skew 500 ppm
Jun 20 19:09:19 dist-test-slave-6wrf krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:19.974690 21874 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:09:19.975703 21874 webserver.cc:460] Webserver started at http://127.0.0.1:44303/ using document root <none> and password file <none>
I20260620 19:09:19.976255 21874 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:19.976346 21874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:19.976573 21874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:19.978647 21874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "52d98c45c2124a648946107c280f1923"
format_stamp: "Formatted at 2026-06-20 19:09:19 on dist-test-slave-6wrf"
server_key: "63601ef51b989b156306dee7e20375ef"
server_key_iv: "e1d9d3a4006f7c0e54b4e0232d0d1371"
server_key_version: "encryptionkey@0"
I20260620 19:09:19.979192 21874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "52d98c45c2124a648946107c280f1923"
format_stamp: "Formatted at 2026-06-20 19:09:19 on dist-test-slave-6wrf"
server_key: "63601ef51b989b156306dee7e20375ef"
server_key_iv: "e1d9d3a4006f7c0e54b4e0232d0d1371"
server_key_version: "encryptionkey@0"
I20260620 19:09:19.982918 21874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:09:19.985770 21889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:19.987079 21874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:19.987181 21874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "52d98c45c2124a648946107c280f1923"
format_stamp: "Formatted at 2026-06-20 19:09:19 on dist-test-slave-6wrf"
server_key: "63601ef51b989b156306dee7e20375ef"
server_key_iv: "e1d9d3a4006f7c0e54b4e0232d0d1371"
server_key_version: "encryptionkey@0"
I20260620 19:09:19.987273 21874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:20.017807 21874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:20.021138 21874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:20.021297 21874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:20.028334 21874 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40717
I20260620 19:09:20.028409 21941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40717 every 8 connection(s)
I20260620 19:09:20.029330 21874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:20.032145 21942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:20.032573 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 21874
I20260620 19:09:20.032711 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:20.033013 1642 external_mini_cluster.cc:1428] Setting key 494a34df31b2b13f492cf4cdc8295fc5
I20260620 19:09:20.038391 21942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923: Bootstrap starting.
I20260620 19:09:20.040925 21942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:20.041839 21942 log.cc:824] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:09:20 dist-test-slave-6wrf krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:09:20.044827 21942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923: No bootstrap required, opened a new log
I20260620 19:09:20.047433 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:20.034497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44187 (local address 127.0.0.1:40717)
0620 19:09:20.034908 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:20.034923 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:20.034969 (+ 46us) server_negotiation.cc:407] Connection header received
0620 19:09:20.035594 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:20.035609 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:20.035919 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:20.036191 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:20.037121 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:20.038152 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:20.039282 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:20.039733 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:20.039883 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:20.042819 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:20.042824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:20.042836 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:20.042865 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:20.044565 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:20.045143 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:20.045148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:20.045153 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:20.045255 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:20.045622 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:20.045626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:20.045627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:20.046013 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:20.046243 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:20.046502 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 19:09:20.046719 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":139,"threads_started":1}
I20260620 19:09:20.049206 21942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d98c45c2124a648946107c280f1923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } }
I20260620 19:09:20.049501 21942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:20.049551 21942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d98c45c2124a648946107c280f1923, State: Initialized, Role: FOLLOWER
I20260620 19:09:20.050014 21942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d98c45c2124a648946107c280f1923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } }
I20260620 19:09:20.050153 21942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:20.050235 21942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:20.050387 21942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:20.051103 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:20.051550 21942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d98c45c2124a648946107c280f1923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } }
I20260620 19:09:20.051928 21942 leader_election.cc:302] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52d98c45c2124a648946107c280f1923; no voters:
I20260620 19:09:20.052300 21942 leader_election.cc:288] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:20.052484 21947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:20.052748 21947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [term 1 LEADER]: Becoming Leader. State: Replica: 52d98c45c2124a648946107c280f1923, State: Running, Role: LEADER
I20260620 19:09:20.053109 21947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d98c45c2124a648946107c280f1923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } }
I20260620 19:09:20.053843 21942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:20.054472 21953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52d98c45c2124a648946107c280f1923. Latest consensus state: current_term: 1 leader_uuid: "52d98c45c2124a648946107c280f1923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d98c45c2124a648946107c280f1923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } } }
I20260620 19:09:20.054622 21953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:20.054895 21952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52d98c45c2124a648946107c280f1923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d98c45c2124a648946107c280f1923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } } }
I20260620 19:09:20.054981 21952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:20.055161 21960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:20.058437 21960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:20.063748 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:20.053685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45193 (local address 127.0.0.1:40717)
0620 19:09:20.053805 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:20.053809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:20.053823 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:09:20.053863 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:20.053866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:20.053914 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:20.053991 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:20.056329 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:20.056857 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:20.058041 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:20.058400 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:20.058497 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:20.059937 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:20.059942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:20.059945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:20.059975 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:20.061458 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:20.062061 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:20.062064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:20.062066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:20.062149 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:20.062513 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:20.062516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:20.062518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:20.062720 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:20.062863 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:20.063554 (+ 691us) server_negotiation.cc:299] Negotiation successful
0620 19:09:20.063638 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:09:20.065968 21960 catalog_manager.cc:1347] Generated new cluster ID: d22cb48cc11b481e8793e338da31bfb1
I20260620 19:09:20.066061 21960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:20.089340 21960 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:20.090809 21960 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:20.099941 21960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52d98c45c2124a648946107c280f1923: Generated new TSK 0
I20260620 19:09:20.100782 21960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:20.113427 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 21874
W20260620 19:09:20.119138 21859 connection.cc:535] client connection to 127.0.0.1:40717 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40717): Connection reset by peer (error 104)
2026-06-20T19:09:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1490 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:20 dist-test-slave-6wrf krb5kdc[21973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:20 dist-test-slave-6wrf krb5kdc[21973](info): set up 2 sockets
Jun 20 19:09:20 dist-test-slave-6wrf krb5kdc[21973](info): commencing operation
krb5kdc: starting...
W20260620 19:09:21.174665 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:21 dist-test-slave-6wrf krb5kdc[21973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:09:21.309026 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:21.420445 21986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:21.420722 21986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:21.420764 21986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:21.424873 21986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:21.424945 21986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:21.424964 21986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:21.424981 21986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:21.424996 21986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:21.430691 21986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:21.431846 21986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:21.432861 21986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:21.441933 21992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:21.442080 21994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:21.442094 21991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:21.442726 21986 server_base.cc:1034] running on GCE node
I20260620 19:09:21.443303 21986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:21.444407 21986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:21.445696 21986 hybrid_clock.cc:648] HybridClock initialized: now 1781982561445655 us; error 53 us; skew 500 ppm
Jun 20 19:09:21 dist-test-slave-6wrf krb5kdc[21973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:21.449115 21986 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:09:21.450738 21986 webserver.cc:460] Webserver started at http://127.0.0.1:33887/ using document root <none> and password file <none>
I20260620 19:09:21.451601 21986 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:21.451750 21986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:21.452011 21986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:21.454157 21986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "a338c22ed5074b4abbf1668b8af627eb"
format_stamp: "Formatted at 2026-06-20 19:09:21 on dist-test-slave-6wrf"
server_key: "e0e46766947c0bda242099a14ca35c45"
server_key_iv: "e534bb9fce48fb1e4ef2adb1e406c70f"
server_key_version: "encryptionkey@0"
I20260620 19:09:21.454727 21986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "a338c22ed5074b4abbf1668b8af627eb"
format_stamp: "Formatted at 2026-06-20 19:09:21 on dist-test-slave-6wrf"
server_key: "e0e46766947c0bda242099a14ca35c45"
server_key_iv: "e534bb9fce48fb1e4ef2adb1e406c70f"
server_key_version: "encryptionkey@0"
I20260620 19:09:21.458485 21986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 19:09:21.461254 22001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:21.462971 21986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:09:21.463128 21986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "a338c22ed5074b4abbf1668b8af627eb"
format_stamp: "Formatted at 2026-06-20 19:09:21 on dist-test-slave-6wrf"
server_key: "e0e46766947c0bda242099a14ca35c45"
server_key_iv: "e534bb9fce48fb1e4ef2adb1e406c70f"
server_key_version: "encryptionkey@0"
I20260620 19:09:21.463253 21986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:21.476935 21986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:21.480396 21986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:21.480607 21986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:21.488622 21986 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42299
I20260620 19:09:21.488652 22053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42299 every 8 connection(s)
I20260620 19:09:21.489701 21986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:21.492406 22054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:21.496441 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 21986
I20260620 19:09:21.496548 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:21.496850 1642 external_mini_cluster.cc:1428] Setting key cace4d4cbe5621f00e0ab38b6689766f
I20260620 19:09:21.498167 22054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb: Bootstrap starting.
I20260620 19:09:21.500818 22054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:21.501573 22054 log.cc:824] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:21.503782 22054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb: No bootstrap required, opened a new log
I20260620 19:09:21.507303 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:21.498361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52531 (local address 127.0.0.1:42299)
0620 19:09:21.498787 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:21.498805 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:21.498845 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:09:21.499686 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:21.499706 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:21.500159 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:21.500504 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:21.501167 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:21.502626 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:21.503761 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:21.504436 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:21.504663 (+ 227us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:21.505685 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:21.505690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:21.505703 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:21.506081 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:21.506131 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:21.506156 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:21.506301 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:21.506468 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 19:09:21.506605 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":107,"threads_started":1}
I20260620 19:09:21.508415 22054 raft_consensus.cc:348] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a338c22ed5074b4abbf1668b8af627eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42299 } }
I20260620 19:09:21.508706 22054 raft_consensus.cc:374] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:21.508761 22054 raft_consensus.cc:729] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a338c22ed5074b4abbf1668b8af627eb, State: Initialized, Role: FOLLOWER
I20260620 19:09:21.509231 22054 consensus_queue.cc:260] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a338c22ed5074b4abbf1668b8af627eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42299 } }
I20260620 19:09:21.509408 22054 raft_consensus.cc:388] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:21.509475 22054 raft_consensus.cc:482] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:21.509629 22054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:21.510720 22054 raft_consensus.cc:504] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a338c22ed5074b4abbf1668b8af627eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42299 } }
I20260620 19:09:21.510912 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:21.511090 22054 leader_election.cc:302] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a338c22ed5074b4abbf1668b8af627eb; no voters:
I20260620 19:09:21.511442 22054 leader_election.cc:288] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:21.511510 22059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:21.511703 22059 raft_consensus.cc:686] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [term 1 LEADER]: Becoming Leader. State: Replica: a338c22ed5074b4abbf1668b8af627eb, State: Running, Role: LEADER
I20260620 19:09:21.512107 22059 consensus_queue.cc:237] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a338c22ed5074b4abbf1668b8af627eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42299 } }
I20260620 19:09:21.512727 22054 sys_catalog.cc:564] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:21.513412 22059 sys_catalog.cc:455] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a338c22ed5074b4abbf1668b8af627eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a338c22ed5074b4abbf1668b8af627eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42299 } } }
I20260620 19:09:21.513548 22059 sys_catalog.cc:458] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:21.514072 22068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:21.513978 22060 sys_catalog.cc:455] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a338c22ed5074b4abbf1668b8af627eb. Latest consensus state: current_term: 1 leader_uuid: "a338c22ed5074b4abbf1668b8af627eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a338c22ed5074b4abbf1668b8af627eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42299 } } }
I20260620 19:09:21.514966 22060 sys_catalog.cc:458] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:21.518340 22068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:21.521104 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:21.514573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51313 (local address 127.0.0.1:42299)
0620 19:09:21.514899 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:21.514907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:21.514992 (+ 85us) server_negotiation.cc:407] Connection header received
0620 19:09:21.515238 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:21.515242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:21.515305 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:21.515416 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:21.515795 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:21.516634 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:21.518026 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:21.518404 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:21.518519 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:21.520377 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:21.520381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:21.520384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:21.520399 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:21.520432 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:21.520446 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:21.520539 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:21.520905 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 19:09:21.520943 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260620 19:09:21.526443 22068 catalog_manager.cc:1347] Generated new cluster ID: a58bc34790864812960be1a149d2281b
I20260620 19:09:21.526538 22068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:21.539268 22068 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:21.540215 22068 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:21.551522 22068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a338c22ed5074b4abbf1668b8af627eb: Generated new TSK 0
I20260620 19:09:21.552241 22068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:21.563894 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 21986
W20260620 19:09:21.569479 21971 connection.cc:535] client connection to 127.0.0.1:42299 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42299): Connection reset by peer (error 104)
2026-06-20T19:09:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:21 dist-test-slave-6wrf krb5kdc[22085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:21 dist-test-slave-6wrf krb5kdc[22085](info): set up 2 sockets
Jun 20 19:09:21 dist-test-slave-6wrf krb5kdc[22085](info): commencing operation
krb5kdc: starting...
W20260620 19:09:22.638304 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:22 dist-test-slave-6wrf krb5kdc[22085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:09:22.773102 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35293
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:22.888674 22098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:22.889003 22098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:22.889083 22098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:22.893239 22098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:22.893348 22098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:22.893386 22098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:22.893424 22098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:22.893456 22098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:22.898916 22098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35293
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:22.900060 22098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:22.901095 22098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:22.910130 22106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:22.910207 22104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:22.910300 22103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:22.910422 22098 server_base.cc:1034] running on GCE node
I20260620 19:09:22.911518 22098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:22.913187 22098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:22.914592 22098 hybrid_clock.cc:648] HybridClock initialized: now 1781982562914546 us; error 81 us; skew 500 ppm
Jun 20 19:09:22 dist-test-slave-6wrf krb5kdc[22085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:22.917797 22098 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:09:22.918844 22098 webserver.cc:460] Webserver started at http://127.0.0.1:36417/ using document root <none> and password file <none>
I20260620 19:09:22.919469 22098 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:22.919562 22098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:22.919803 22098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:22.921988 22098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "0784fdff5dae46a7b60693e372dadbb5"
format_stamp: "Formatted at 2026-06-20 19:09:22 on dist-test-slave-6wrf"
server_key: "fe558617ee83fe7690af38de5afd8466"
server_key_iv: "80532b0e8273a0aba25026f4a1b89350"
server_key_version: "encryptionkey@0"
I20260620 19:09:22.922560 22098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "0784fdff5dae46a7b60693e372dadbb5"
format_stamp: "Formatted at 2026-06-20 19:09:22 on dist-test-slave-6wrf"
server_key: "fe558617ee83fe7690af38de5afd8466"
server_key_iv: "80532b0e8273a0aba25026f4a1b89350"
server_key_version: "encryptionkey@0"
I20260620 19:09:22.926348 22098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:09:22.928975 22113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:22.930485 22098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:22.930632 22098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "0784fdff5dae46a7b60693e372dadbb5"
format_stamp: "Formatted at 2026-06-20 19:09:22 on dist-test-slave-6wrf"
server_key: "fe558617ee83fe7690af38de5afd8466"
server_key_iv: "80532b0e8273a0aba25026f4a1b89350"
server_key_version: "encryptionkey@0"
I20260620 19:09:22.930756 22098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:22.935436 22098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:22.938642 22098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:22.938844 22098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:22.945886 22098 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35293
I20260620 19:09:22.946022 22165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35293 every 8 connection(s)
I20260620 19:09:22.946830 22098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:22.949699 22166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:22.950284 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 22098
I20260620 19:09:22.950428 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:22.950739 1642 external_mini_cluster.cc:1428] Setting key d47fac3dc4a9d45cba8512f470d7ae4c
I20260620 19:09:22.956265 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:22.952138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57907 (local address 127.0.0.1:35293)
0620 19:09:22.952579 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:22.952597 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:22.952641 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:09:22.953433 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:22.953453 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:22.953815 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:22.954164 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:22.954966 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:22.954972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:22.954987 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:22.954999 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:22.955063 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:22.955093 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:22.955232 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:22.955296 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 19:09:22.955438 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":131,"threads_started":1}
I20260620 19:09:22.956292 22166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5: Bootstrap starting.
I20260620 19:09:22.958694 22166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:22.959537 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:22.959539 22166 log.cc:824] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:22.962738 22166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5: No bootstrap required, opened a new log
I20260620 19:09:22.964421 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:22.962721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40037 (local address 127.0.0.1:35293)
0620 19:09:22.962861 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:22.962865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:22.963110 (+ 245us) server_negotiation.cc:407] Connection header received
0620 19:09:22.963217 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:22.963219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:22.963250 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:22.963323 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:22.964033 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:22.964036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:22.964038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:22.964047 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:22.964069 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:22.964078 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:22.964147 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:22.964241 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 19:09:22.964270 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 19:09:22.967195 22166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0784fdff5dae46a7b60693e372dadbb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } }
I20260620 19:09:22.967456 22166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:22.967517 22166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0784fdff5dae46a7b60693e372dadbb5, State: Initialized, Role: FOLLOWER
I20260620 19:09:22.967976 22166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0784fdff5dae46a7b60693e372dadbb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } }
I20260620 19:09:22.968128 22166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:22.968199 22166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:22.968298 22166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:22.969250 22166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0784fdff5dae46a7b60693e372dadbb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } }
I20260620 19:09:22.969640 22166 leader_election.cc:302] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0784fdff5dae46a7b60693e372dadbb5; no voters:
I20260620 19:09:22.970001 22166 leader_election.cc:288] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:22.970065 22176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:22.970270 22176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [term 1 LEADER]: Becoming Leader. State: Replica: 0784fdff5dae46a7b60693e372dadbb5, State: Running, Role: LEADER
I20260620 19:09:22.970567 22176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0784fdff5dae46a7b60693e372dadbb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } }
I20260620 19:09:22.971328 22166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:22.972455 22177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0784fdff5dae46a7b60693e372dadbb5. Latest consensus state: current_term: 1 leader_uuid: "0784fdff5dae46a7b60693e372dadbb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0784fdff5dae46a7b60693e372dadbb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } } }
I20260620 19:09:22.972666 22177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:22.972601 22178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0784fdff5dae46a7b60693e372dadbb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0784fdff5dae46a7b60693e372dadbb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } } }
I20260620 19:09:22.972874 22178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:22.973881 22185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:22.977293 22185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:22.982878 22185 catalog_manager.cc:1347] Generated new cluster ID: 6d6c460ae13d4fdda99c2be293cdac6b
I20260620 19:09:22.982995 22185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:23.005198 22185 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:23.005564 22185 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:23.010569 22185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0784fdff5dae46a7b60693e372dadbb5: Generated new TSK 0
I20260620 19:09:23.011224 22185 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:23.023885 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 22098
2026-06-20T19:09:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:23 dist-test-slave-6wrf krb5kdc[22197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:23 dist-test-slave-6wrf krb5kdc[22197](info): set up 2 sockets
Jun 20 19:09:23 dist-test-slave-6wrf krb5kdc[22197](info): commencing operation
krb5kdc: starting...
W20260620 19:09:24.106482 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:24 dist-test-slave-6wrf krb5kdc[22197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:09:24.239322 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41957
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:24.363613 22210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:24.363906 22210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:24.363951 22210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:24.369457 22210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:24.369567 22210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:24.369591 22210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:24.369680 22210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:24.369714 22210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:24.376034 22210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:24.377216 22210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:24.378274 22210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:24.387883 22215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:24.387998 22210 server_base.cc:1034] running on GCE node
W20260620 19:09:24.387883 22218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:24.387903 22216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:24.388777 22210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:24.389983 22210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:24.391178 22210 hybrid_clock.cc:648] HybridClock initialized: now 1781982564391164 us; error 53 us; skew 500 ppm
Jun 20 19:09:24 dist-test-slave-6wrf krb5kdc[22197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:24.398802 22210 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:09:24.399856 22210 webserver.cc:460] Webserver started at http://127.0.0.1:34763/ using document root <none> and password file <none>
I20260620 19:09:24.400444 22210 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:24.400537 22210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:24.400769 22210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:24.402889 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "9f396113563d4a58820d874434879c4a"
format_stamp: "Formatted at 2026-06-20 19:09:24 on dist-test-slave-6wrf"
server_key: "4cd7bc94a320352c18bfc54526b70ebf"
server_key_iv: "992b23460a5f728cfac6feec27142b3b"
server_key_version: "encryptionkey@0"
I20260620 19:09:24.403452 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "9f396113563d4a58820d874434879c4a"
format_stamp: "Formatted at 2026-06-20 19:09:24 on dist-test-slave-6wrf"
server_key: "4cd7bc94a320352c18bfc54526b70ebf"
server_key_iv: "992b23460a5f728cfac6feec27142b3b"
server_key_version: "encryptionkey@0"
I20260620 19:09:24.407240 22210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:09:24.409825 22225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:24.410979 22210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:09:24.411141 22210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "9f396113563d4a58820d874434879c4a"
format_stamp: "Formatted at 2026-06-20 19:09:24 on dist-test-slave-6wrf"
server_key: "4cd7bc94a320352c18bfc54526b70ebf"
server_key_iv: "992b23460a5f728cfac6feec27142b3b"
server_key_version: "encryptionkey@0"
I20260620 19:09:24.411268 22210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:24.422727 22210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:24.425837 22210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:24.426041 22210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:24.433300 22210 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41957
I20260620 19:09:24.433352 22277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41957 every 8 connection(s)
I20260620 19:09:24.434376 22210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:24.437093 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 22210
I20260620 19:09:24.437359 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:24.437821 1642 external_mini_cluster.cc:1428] Setting key 66fd96be890a1f063295ef6f0c9d2495
I20260620 19:09:24.438213 22278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:24.445176 22278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a: Bootstrap starting.
I20260620 19:09:24.447721 22278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a: Neither blocks nor log segments found. Creating new log.
Jun 20 19:09:24 dist-test-slave-6wrf krb5kdc[22197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:09:24.448587 22278 log.cc:824] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:24.451013 22278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a: No bootstrap required, opened a new log
I20260620 19:09:24.453516 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:24.439728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52285 (local address 127.0.0.1:41957)
0620 19:09:24.440189 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:24.440205 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:24.440240 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:09:24.440894 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:24.440911 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:24.441252 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:24.441517 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:24.442528 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:24.443635 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:24.444897 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:24.445452 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:24.445683 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:24.448445 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:24.448450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:24.448463 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:24.448491 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:24.450492 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:24.451026 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:24.451031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:24.451036 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:24.451217 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:24.451546 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:24.451548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:24.451549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:24.451940 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:24.452089 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:24.452417 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 19:09:24.452821 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":164,"threads_started":1}
I20260620 19:09:24.455266 22278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f396113563d4a58820d874434879c4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41957 } }
I20260620 19:09:24.455556 22278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:24.455603 22278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f396113563d4a58820d874434879c4a, State: Initialized, Role: FOLLOWER
I20260620 19:09:24.456049 22278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f396113563d4a58820d874434879c4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41957 } }
I20260620 19:09:24.456182 22278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:24.456271 22278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:24.456403 22278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:24.457125 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:24.457782 22278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f396113563d4a58820d874434879c4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41957 } }
I20260620 19:09:24.458135 22278 leader_election.cc:302] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f396113563d4a58820d874434879c4a; no voters:
I20260620 19:09:24.458575 22278 leader_election.cc:288] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:24.458604 22285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:24.458881 22285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [term 1 LEADER]: Becoming Leader. State: Replica: 9f396113563d4a58820d874434879c4a, State: Running, Role: LEADER
I20260620 19:09:24.459223 22285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f396113563d4a58820d874434879c4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41957 } }
I20260620 19:09:24.460000 22278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:24.460748 22289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f396113563d4a58820d874434879c4a. Latest consensus state: current_term: 1 leader_uuid: "9f396113563d4a58820d874434879c4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f396113563d4a58820d874434879c4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41957 } } }
I20260620 19:09:24.460958 22289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:24.461319 22288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f396113563d4a58820d874434879c4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f396113563d4a58820d874434879c4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41957 } } }
I20260620 19:09:24.461467 22288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:24.461783 22296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:24.465516 22296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:24.474412 22296 catalog_manager.cc:1347] Generated new cluster ID: 2036145f63904223b70a101e7ad7a0ec
I20260620 19:09:24.474530 22296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:24.479487 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:24.460611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39523 (local address 127.0.0.1:41957)
0620 19:09:24.462597 (+ 1986us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:24.462602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:24.462898 (+ 296us) server_negotiation.cc:407] Connection header received
0620 19:09:24.463157 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:24.463164 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:24.463286 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:24.463764 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:24.465084 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:24.467048 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:24.468590 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:24.468940 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:24.469502 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:24.474792 (+ 5290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:24.474797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:24.474801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:24.474842 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:24.477319 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:24.477995 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:24.478000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:24.478002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:24.478112 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:24.478437 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:24.478441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:24.478443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:24.478707 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:24.478870 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:24.479132 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 19:09:24.479278 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1888}
I20260620 19:09:24.485762 22296 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:24.487409 22296 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:24.501902 22296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f396113563d4a58820d874434879c4a: Generated new TSK 0
I20260620 19:09:24.502740 22296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:24.508808 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 22210
2026-06-20T19:09:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1484 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:24 dist-test-slave-6wrf krb5kdc[22309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:24 dist-test-slave-6wrf krb5kdc[22309](info): set up 2 sockets
Jun 20 19:09:24 dist-test-slave-6wrf krb5kdc[22309](info): commencing operation
krb5kdc: starting...
W20260620 19:09:25.585922 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:25 dist-test-slave-6wrf krb5kdc[22309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:09:25.712873 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:25.825030 22322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:25.825362 22322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:25.825457 22322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:25.830511 22322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:25.830688 22322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:25.830732 22322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:25.830776 22322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:25.830813 22322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:25.837208 22322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:25.838356 22322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:25.839373 22322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:09:25.848181 22322 server_base.cc:1034] running on GCE node
W20260620 19:09:25.848177 22330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:25.848315 22327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:25.848315 22328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:25.849133 22322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:25.850239 22322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:25.851423 22322 hybrid_clock.cc:648] HybridClock initialized: now 1781982565851407 us; error 33 us; skew 500 ppm
Jun 20 19:09:25 dist-test-slave-6wrf krb5kdc[22309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:25.854676 22322 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:09:25.855684 22322 webserver.cc:460] Webserver started at http://127.0.0.1:33917/ using document root <none> and password file <none>
I20260620 19:09:25.856236 22322 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:25.856307 22322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:25.856547 22322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:25.858680 22322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "c84997a961f647fdb7ae229987341e58"
format_stamp: "Formatted at 2026-06-20 19:09:25 on dist-test-slave-6wrf"
server_key: "0eb888aba892cb46423891497cdb962b"
server_key_iv: "570095fb41ca6774c80489cc0f9a1cae"
server_key_version: "encryptionkey@0"
I20260620 19:09:25.859372 22322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "c84997a961f647fdb7ae229987341e58"
format_stamp: "Formatted at 2026-06-20 19:09:25 on dist-test-slave-6wrf"
server_key: "0eb888aba892cb46423891497cdb962b"
server_key_iv: "570095fb41ca6774c80489cc0f9a1cae"
server_key_version: "encryptionkey@0"
I20260620 19:09:25.863832 22322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:09:25.867121 22337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:25.868417 22322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:09:25.868577 22322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "c84997a961f647fdb7ae229987341e58"
format_stamp: "Formatted at 2026-06-20 19:09:25 on dist-test-slave-6wrf"
server_key: "0eb888aba892cb46423891497cdb962b"
server_key_iv: "570095fb41ca6774c80489cc0f9a1cae"
server_key_version: "encryptionkey@0"
I20260620 19:09:25.868708 22322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:25.890527 22322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:25.894193 22322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:25.894429 22322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:25.901875 22322 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38305
I20260620 19:09:25.901952 22389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38305 every 8 connection(s)
I20260620 19:09:25.902894 22322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:25.905650 22390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:25.910137 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 22322
I20260620 19:09:25.910310 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:25.910622 1642 external_mini_cluster.cc:1428] Setting key 2492a28182b8e16c6812bb6356f1bc01
I20260620 19:09:25.911247 22390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58: Bootstrap starting.
I20260620 19:09:25.913547 22390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:25.914296 22390 log.cc:824] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:25.916116 22390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58: No bootstrap required, opened a new log
I20260620 19:09:25.919911 22390 raft_consensus.cc:348] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84997a961f647fdb7ae229987341e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260620 19:09:25.920159 22390 raft_consensus.cc:374] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:25.920244 22390 raft_consensus.cc:729] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84997a961f647fdb7ae229987341e58, State: Initialized, Role: FOLLOWER
I20260620 19:09:25.920487 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:25.911960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55521 (local address 127.0.0.1:38305)
0620 19:09:25.912385 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:25.912399 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:25.912432 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:09:25.913056 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:25.913071 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:25.913390 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:25.913742 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:25.914138 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:25.915217 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:25.916103 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:25.916610 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:25.916787 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:25.917683 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:25.917688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:25.917700 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:25.918090 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:25.918304 (+ 214us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:25.918378 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:25.918613 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:25.918884 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 19:09:25.919274 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":123,"threads_started":1}
I20260620 19:09:25.920747 22390 consensus_queue.cc:260] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84997a961f647fdb7ae229987341e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260620 19:09:25.920919 22390 raft_consensus.cc:388] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:25.920977 22390 raft_consensus.cc:482] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:25.921069 22390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:25.922077 22390 raft_consensus.cc:504] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84997a961f647fdb7ae229987341e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260620 19:09:25.922448 22390 leader_election.cc:302] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c84997a961f647fdb7ae229987341e58; no voters:
I20260620 19:09:25.922827 22390 leader_election.cc:288] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:25.922927 22395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:25.923148 22395 raft_consensus.cc:686] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [term 1 LEADER]: Becoming Leader. State: Replica: c84997a961f647fdb7ae229987341e58, State: Running, Role: LEADER
I20260620 19:09:25.923492 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:25.924034 22390 sys_catalog.cc:564] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:25.925348 22395 consensus_queue.cc:237] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84997a961f647fdb7ae229987341e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260620 19:09:25.927001 22396 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c84997a961f647fdb7ae229987341e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84997a961f647fdb7ae229987341e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } } }
I20260620 19:09:25.927986 22396 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:25.927174 22397 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c84997a961f647fdb7ae229987341e58. Latest consensus state: current_term: 1 leader_uuid: "c84997a961f647fdb7ae229987341e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84997a961f647fdb7ae229987341e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } } }
I20260620 19:09:25.928265 22397 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58 [sys.catalog]: This master's current role is: LEADER
W20260620 19:09:25.929711 22414 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:09:25.929814 22414 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:09:25.929919 22416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:25.930521 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:25.925531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44681 (local address 127.0.0.1:38305)
0620 19:09:25.925681 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:25.925688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:25.926227 (+ 539us) server_negotiation.cc:407] Connection header received
0620 19:09:25.926380 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:25.926383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:25.926433 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:25.926527 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:25.926828 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:25.927533 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:25.928817 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:25.929044 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:25.929177 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:25.930056 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:25.930060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:25.930063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:25.930077 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:25.930106 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:25.930117 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:25.930208 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:25.930330 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 19:09:25.930389 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:09:25.930799 22416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:25.935967 22416 catalog_manager.cc:1347] Generated new cluster ID: b8eb750b6bdd43a08811a3a20a3be673
I20260620 19:09:25.936077 22416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:25.952728 22416 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:25.953851 22416 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:25.960155 22416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c84997a961f647fdb7ae229987341e58: Generated new TSK 0
I20260620 19:09:25.960824 22416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:25.968181 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 22322
2026-06-20T19:09:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1458 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:26 dist-test-slave-6wrf krb5kdc[22421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:26 dist-test-slave-6wrf krb5kdc[22421](info): set up 2 sockets
Jun 20 19:09:26 dist-test-slave-6wrf krb5kdc[22421](info): commencing operation
krb5kdc: starting...
W20260620 19:09:27.030115 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:27 dist-test-slave-6wrf krb5kdc[22421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:09:27.161582 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:27.284510 22434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:27.284817 22434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:27.284864 22434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:27.289263 22434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:27.289326 22434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:27.289346 22434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:27.289363 22434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:27.289378 22434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:27.294854 22434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:27.295957 22434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:27.297056 22434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:27.306516 22442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:27.306461 22440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:27.306452 22439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:27.306913 22434 server_base.cc:1034] running on GCE node
I20260620 19:09:27.307657 22434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:27.308926 22434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:27.310148 22434 hybrid_clock.cc:648] HybridClock initialized: now 1781982567310132 us; error 35 us; skew 500 ppm
Jun 20 19:09:27 dist-test-slave-6wrf krb5kdc[22421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:27.313107 22434 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:09:27.314065 22434 webserver.cc:460] Webserver started at http://127.0.0.1:45271/ using document root <none> and password file <none>
I20260620 19:09:27.314620 22434 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:27.314709 22434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:27.314929 22434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:27.316928 22434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "dd6493fb6a124fb99962344874502159"
format_stamp: "Formatted at 2026-06-20 19:09:27 on dist-test-slave-6wrf"
server_key: "8ad6057f55ac45c007692f4a0e7cfe45"
server_key_iv: "ab80c32f60d798c458fcffe01fafe312"
server_key_version: "encryptionkey@0"
I20260620 19:09:27.317504 22434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "dd6493fb6a124fb99962344874502159"
format_stamp: "Formatted at 2026-06-20 19:09:27 on dist-test-slave-6wrf"
server_key: "8ad6057f55ac45c007692f4a0e7cfe45"
server_key_iv: "ab80c32f60d798c458fcffe01fafe312"
server_key_version: "encryptionkey@0"
I20260620 19:09:27.321264 22434 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:27.324718 22449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:27.326596 22434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 19:09:27.326788 22434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "dd6493fb6a124fb99962344874502159"
format_stamp: "Formatted at 2026-06-20 19:09:27 on dist-test-slave-6wrf"
server_key: "8ad6057f55ac45c007692f4a0e7cfe45"
server_key_iv: "ab80c32f60d798c458fcffe01fafe312"
server_key_version: "encryptionkey@0"
I20260620 19:09:27.326926 22434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:27.332127 22434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:27.335512 22434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:27.335723 22434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:27.345139 22434 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33327
I20260620 19:09:27.345315 22501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33327 every 8 connection(s)
I20260620 19:09:27.346251 22434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:27.348843 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 22434
I20260620 19:09:27.349030 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:27.349335 1642 external_mini_cluster.cc:1428] Setting key a0fc2f557f866fea2d4305602456d46f
I20260620 19:09:27.349501 22502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:27.355516 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:27.350888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37031 (local address 127.0.0.1:33327)
0620 19:09:27.351345 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:27.351365 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:27.351411 (+ 46us) server_negotiation.cc:407] Connection header received
0620 19:09:27.352483 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:27.352507 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:27.352859 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:27.353234 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:27.354100 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:27.354106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:27.354120 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:27.354133 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:27.354196 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:27.354222 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:27.354362 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:27.354470 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 19:09:27.354615 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":117,"threads_started":1}
I20260620 19:09:27.357020 22502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159: Bootstrap starting.
I20260620 19:09:27.358582 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:27.359483 22502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:27.360239 22502 log.cc:824] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:27.362849 22502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159: No bootstrap required, opened a new log
I20260620 19:09:27.362879 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:27.360672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52143 (local address 127.0.0.1:33327)
0620 19:09:27.360783 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:27.360787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:27.361183 (+ 396us) server_negotiation.cc:407] Connection header received
0620 19:09:27.361289 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:27.361298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:27.361356 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:27.361499 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:27.362433 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:27.362437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:27.362440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:27.362452 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:27.362484 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:27.362495 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:27.362580 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:27.362696 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 19:09:27.362729 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:09:27.366398 22502 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6493fb6a124fb99962344874502159" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } }
I20260620 19:09:27.366654 22502 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:27.366736 22502 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6493fb6a124fb99962344874502159, State: Initialized, Role: FOLLOWER
I20260620 19:09:27.367149 22502 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6493fb6a124fb99962344874502159" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } }
I20260620 19:09:27.367298 22502 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:27.367385 22502 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:27.367479 22502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:27.368371 22502 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6493fb6a124fb99962344874502159" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } }
I20260620 19:09:27.368713 22502 leader_election.cc:302] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd6493fb6a124fb99962344874502159; no voters:
I20260620 19:09:27.369050 22502 leader_election.cc:288] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:27.369192 22512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:27.369483 22512 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [term 1 LEADER]: Becoming Leader. State: Replica: dd6493fb6a124fb99962344874502159, State: Running, Role: LEADER
I20260620 19:09:27.369874 22512 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6493fb6a124fb99962344874502159" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } }
I20260620 19:09:27.370282 22502 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:27.371738 22513 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd6493fb6a124fb99962344874502159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6493fb6a124fb99962344874502159" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } } }
I20260620 19:09:27.371924 22513 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:27.372426 22521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:27.373342 22514 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd6493fb6a124fb99962344874502159. Latest consensus state: current_term: 1 leader_uuid: "dd6493fb6a124fb99962344874502159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6493fb6a124fb99962344874502159" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } } }
I20260620 19:09:27.373446 22514 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:27.375918 22521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:27.381812 22521 catalog_manager.cc:1347] Generated new cluster ID: 35162ffec93242639a790eb8a3d495f2
I20260620 19:09:27.381932 22521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:27.396497 22521 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:27.396873 22521 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:27.404009 22521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd6493fb6a124fb99962344874502159: Generated new TSK 0
I20260620 19:09:27.404709 22521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:27.410718 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 22434
2026-06-20T19:09:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:27 dist-test-slave-6wrf krb5kdc[22533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:27 dist-test-slave-6wrf krb5kdc[22533](info): set up 2 sockets
Jun 20 19:09:27 dist-test-slave-6wrf krb5kdc[22533](info): commencing operation
krb5kdc: starting...
W20260620 19:09:28.475706 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:28 dist-test-slave-6wrf krb5kdc[22533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:09:28.613813 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42599
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:28.731990 22546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:28.732285 22546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:28.732360 22546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:28.736367 22546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:28.736474 22546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:28.736510 22546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:28.736568 22546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:28.736606 22546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:28.741855 22546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:28.742929 22546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:28.743942 22546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:28.752996 22554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:28.753007 22552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:28.753017 22551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:28.753244 22546 server_base.cc:1034] running on GCE node
I20260620 19:09:28.754022 22546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:28.755089 22546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:28.756335 22546 hybrid_clock.cc:648] HybridClock initialized: now 1781982568756301 us; error 37 us; skew 500 ppm
Jun 20 19:09:28 dist-test-slave-6wrf krb5kdc[22533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:28.760396 22546 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:09:28.761886 22546 webserver.cc:460] Webserver started at http://127.0.0.1:40331/ using document root <none> and password file <none>
I20260620 19:09:28.762493 22546 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:28.762588 22546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:28.762835 22546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:28.765041 22546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "fd6c8ad57c92490492c67793b8999717"
format_stamp: "Formatted at 2026-06-20 19:09:28 on dist-test-slave-6wrf"
server_key: "c033d81f088a19e3b0ce182e4fd4366e"
server_key_iv: "6e5bec72eb9d378aee0fbf153bffba04"
server_key_version: "encryptionkey@0"
I20260620 19:09:28.765718 22546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "fd6c8ad57c92490492c67793b8999717"
format_stamp: "Formatted at 2026-06-20 19:09:28 on dist-test-slave-6wrf"
server_key: "c033d81f088a19e3b0ce182e4fd4366e"
server_key_iv: "6e5bec72eb9d378aee0fbf153bffba04"
server_key_version: "encryptionkey@0"
I20260620 19:09:28.769766 22546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:09:28.772778 22561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:28.774503 22546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:28.774672 22546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "fd6c8ad57c92490492c67793b8999717"
format_stamp: "Formatted at 2026-06-20 19:09:28 on dist-test-slave-6wrf"
server_key: "c033d81f088a19e3b0ce182e4fd4366e"
server_key_iv: "6e5bec72eb9d378aee0fbf153bffba04"
server_key_version: "encryptionkey@0"
I20260620 19:09:28.774808 22546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:28.804837 22546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:28.809172 22546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:28.809417 22546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:28.816612 22546 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42599
I20260620 19:09:28.816641 22613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42599 every 8 connection(s)
I20260620 19:09:28.817658 22546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:28.820379 22614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:28.822518 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 22546
I20260620 19:09:28.822661 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:28.823072 1642 external_mini_cluster.cc:1428] Setting key ea19f23522a033c99ae4320465fe1c44
I20260620 19:09:28.827797 22614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717: Bootstrap starting.
I20260620 19:09:28.830433 22614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:28.831283 22614 log.cc:824] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:28.833654 22614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717: No bootstrap required, opened a new log
Jun 20 19:09:28 dist-test-slave-6wrf krb5kdc[22533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:09:28.838135 22614 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6c8ad57c92490492c67793b8999717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42599 } }
I20260620 19:09:28.838418 22614 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:28.838478 22614 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd6c8ad57c92490492c67793b8999717, State: Initialized, Role: FOLLOWER
I20260620 19:09:28.838995 22614 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6c8ad57c92490492c67793b8999717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42599 } }
I20260620 19:09:28.839155 22614 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:28.839213 22614 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:28.839299 22614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:28.840476 22614 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6c8ad57c92490492c67793b8999717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42599 } }
I20260620 19:09:28.840834 22614 leader_election.cc:302] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd6c8ad57c92490492c67793b8999717; no voters:
I20260620 19:09:28.841058 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:28.824568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:50401 (local address 127.0.0.1:42599)
0620 19:09:28.824998 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:28.825018 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:28.825064 (+ 46us) server_negotiation.cc:407] Connection header received
0620 19:09:28.825880 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:28.825902 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:28.826364 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:28.826732 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:28.827440 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:28.828843 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:28.829992 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:28.830730 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:28.830889 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:28.834618 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:28.834624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:28.834640 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:28.834671 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:28.837237 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:28.837813 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:28.837819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:28.837825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:28.837958 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:28.838733 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:28.838736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:28.838738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:28.839245 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:28.839496 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:28.839808 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 19:09:28.840028 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":114,"threads_started":1}
I20260620 19:09:28.841203 22614 leader_election.cc:288] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:28.841413 22619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:28.841722 22619 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [term 1 LEADER]: Becoming Leader. State: Replica: fd6c8ad57c92490492c67793b8999717, State: Running, Role: LEADER
I20260620 19:09:28.842077 22619 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6c8ad57c92490492c67793b8999717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42599 } }
I20260620 19:09:28.842478 22614 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:28.843621 22621 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd6c8ad57c92490492c67793b8999717. Latest consensus state: current_term: 1 leader_uuid: "fd6c8ad57c92490492c67793b8999717" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6c8ad57c92490492c67793b8999717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42599 } } }
I20260620 19:09:28.843768 22621 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:28.844107 22620 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd6c8ad57c92490492c67793b8999717" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6c8ad57c92490492c67793b8999717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42599 } } }
I20260620 19:09:28.844202 22620 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:28.844554 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:28.844641 22628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:28.848443 22628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:28.855455 22628 catalog_manager.cc:1347] Generated new cluster ID: 38ba2985c5b34a489764b908be9df507
I20260620 19:09:28.855613 22628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:28.856529 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:28.846696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:38569 (local address 127.0.0.1:42599)
0620 19:09:28.846815 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:28.846821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:28.847021 (+ 200us) server_negotiation.cc:407] Connection header received
0620 19:09:28.847157 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:28.847160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:28.847210 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:28.847304 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:28.848119 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:28.848864 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:28.849955 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:28.850325 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:28.850440 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:28.852209 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:28.852215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:28.852218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:28.852243 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:28.854115 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:28.854991 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:28.854995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:28.854997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:28.855085 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:28.855527 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:28.855532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:28.855533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:28.855782 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:28.855884 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:28.856234 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 19:09:28.856345 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:09:28.877447 22628 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:28.878423 22628 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:28.882390 22628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd6c8ad57c92490492c67793b8999717: Generated new TSK 0
I20260620 19:09:28.883059 22628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:28.890012 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 22546
W20260620 19:09:28.895445 22531 connection.cc:535] client connection to 127.0.0.1:42599 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42599): Connection reset by peer (error 104)
2026-06-20T19:09:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1478 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:28 dist-test-slave-6wrf krb5kdc[22645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:28 dist-test-slave-6wrf krb5kdc[22645](info): set up 2 sockets
Jun 20 19:09:28 dist-test-slave-6wrf krb5kdc[22645](info): commencing operation
krb5kdc: starting...
W20260620 19:09:29.955049 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:30 dist-test-slave-6wrf krb5kdc[22645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:09:30.084830 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:30.203997 22658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:30.204443 22658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:30.204540 22658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:30.209510 22658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:30.209604 22658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:30.209666 22658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:30.209726 22658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:30.209765 22658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:30.215476 22658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:30.216594 22658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:30.217635 22658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:30.226451 22664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:30.226408 22663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:30.226434 22666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:30.226490 22658 server_base.cc:1034] running on GCE node
I20260620 19:09:30.227268 22658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:30.228431 22658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:30.229801 22658 hybrid_clock.cc:648] HybridClock initialized: now 1781982570229757 us; error 43 us; skew 500 ppm
Jun 20 19:09:30 dist-test-slave-6wrf krb5kdc[22645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:30.236395 22658 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:09:30.237344 22658 webserver.cc:460] Webserver started at http://127.0.0.1:38891/ using document root <none> and password file <none>
I20260620 19:09:30.237993 22658 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:30.238085 22658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:30.238314 22658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:30.240346 22658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "82e938dc5b204f5aa3174ec65a0e84ca"
format_stamp: "Formatted at 2026-06-20 19:09:30 on dist-test-slave-6wrf"
server_key: "29242ae0dd4a0ec0af9ddf19ae60092c"
server_key_iv: "8547eca9e1090b1c756ab34e0afba51a"
server_key_version: "encryptionkey@0"
I20260620 19:09:30.240902 22658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "82e938dc5b204f5aa3174ec65a0e84ca"
format_stamp: "Formatted at 2026-06-20 19:09:30 on dist-test-slave-6wrf"
server_key: "29242ae0dd4a0ec0af9ddf19ae60092c"
server_key_iv: "8547eca9e1090b1c756ab34e0afba51a"
server_key_version: "encryptionkey@0"
I20260620 19:09:30.244632 22658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:09:30.247192 22673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:30.248489 22658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:09:30.248627 22658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "82e938dc5b204f5aa3174ec65a0e84ca"
format_stamp: "Formatted at 2026-06-20 19:09:30 on dist-test-slave-6wrf"
server_key: "29242ae0dd4a0ec0af9ddf19ae60092c"
server_key_iv: "8547eca9e1090b1c756ab34e0afba51a"
server_key_version: "encryptionkey@0"
I20260620 19:09:30.248809 22658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:30.274884 22658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:30.278873 22658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:30.279093 22658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:30.288270 22658 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37243
I20260620 19:09:30.288360 22725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37243 every 8 connection(s)
I20260620 19:09:30.289817 22658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:30.292142 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 22658
I20260620 19:09:30.292303 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:30.292595 1642 external_mini_cluster.cc:1428] Setting key 030e00caf76024ea85b7f533844a2306
I20260620 19:09:30.293139 22726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:30.300238 22726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca: Bootstrap starting.
I20260620 19:09:30.302963 22726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:30.303611 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:30.294268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:48039 (local address 127.0.0.1:37243)
0620 19:09:30.294721 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:30.294740 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:30.294782 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:09:30.295535 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:30.295557 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:30.295995 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:30.296357 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:30.296777 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:30.298273 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:30.299528 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:30.300201 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:30.300378 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:30.301511 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:30.301519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:30.301536 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:30.302054 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:30.302120 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:30.302149 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:30.302288 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:30.302519 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 19:09:30.302643 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":142,"threads_started":1}
I20260620 19:09:30.303826 22726 log.cc:824] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:30.305893 22726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca: No bootstrap required, opened a new log
I20260620 19:09:30.308035 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:30.310212 22726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e938dc5b204f5aa3174ec65a0e84ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } }
I20260620 19:09:30.310508 22726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:30.310590 22726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e938dc5b204f5aa3174ec65a0e84ca, State: Initialized, Role: FOLLOWER
I20260620 19:09:30.311074 22726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e938dc5b204f5aa3174ec65a0e84ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } }
I20260620 19:09:30.311249 22726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:30.311322 22726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:30.311415 22726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:30.312510 22726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e938dc5b204f5aa3174ec65a0e84ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } }
I20260620 19:09:30.312851 22726 leader_election.cc:302] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82e938dc5b204f5aa3174ec65a0e84ca; no voters:
I20260620 19:09:30.313184 22726 leader_election.cc:288] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:30.313285 22736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:30.313539 22736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [term 1 LEADER]: Becoming Leader. State: Replica: 82e938dc5b204f5aa3174ec65a0e84ca, State: Running, Role: LEADER
I20260620 19:09:30.313975 22736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e938dc5b204f5aa3174ec65a0e84ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } }
I20260620 19:09:30.314555 22726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:30.315021 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:30.310278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:34021 (local address 127.0.0.1:37243)
0620 19:09:30.310409 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:30.310413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:30.310611 (+ 198us) server_negotiation.cc:407] Connection header received
0620 19:09:30.310822 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:30.310825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:30.310879 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:30.310972 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:30.311241 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:30.311976 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:30.313054 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:30.313388 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:30.313494 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:30.314476 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:30.314480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:30.314482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:30.314495 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:30.314522 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:30.314534 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:30.314636 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:30.314807 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 19:09:30.314842 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:09:30.315990 22737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82e938dc5b204f5aa3174ec65a0e84ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e938dc5b204f5aa3174ec65a0e84ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } } }
I20260620 19:09:30.316140 22737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:30.316344 22738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82e938dc5b204f5aa3174ec65a0e84ca. Latest consensus state: current_term: 1 leader_uuid: "82e938dc5b204f5aa3174ec65a0e84ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e938dc5b204f5aa3174ec65a0e84ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } } }
I20260620 19:09:30.316533 22738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca [sys.catalog]: This master's current role is: LEADER
W20260620 19:09:30.319660 22751 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:09:30.319749 22751 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:09:30.319914 22750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:30.321025 22750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:30.329005 22750 catalog_manager.cc:1347] Generated new cluster ID: 1897b6b480c447daa9a240c39925ace9
I20260620 19:09:30.329101 22750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:30.344012 22750 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:30.345002 22750 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:30.349465 22750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82e938dc5b204f5aa3174ec65a0e84ca: Generated new TSK 0
I20260620 19:09:30.350217 22750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:30.361488 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 22658
W20260620 19:09:30.366935 22643 connection.cc:535] client connection to 127.0.0.1:37243 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37243): Connection reset by peer (error 104)
2026-06-20T19:09:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1472 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:30 dist-test-slave-6wrf krb5kdc[22757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:30 dist-test-slave-6wrf krb5kdc[22757](info): set up 2 sockets
Jun 20 19:09:30 dist-test-slave-6wrf krb5kdc[22757](info): commencing operation
krb5kdc: starting...
W20260620 19:09:31.426050 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:31 dist-test-slave-6wrf krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:09:31.553411 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:31.669340 22770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:31.669625 22770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:31.669668 22770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:31.673544 22770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:31.673626 22770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:31.673647 22770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:31.673663 22770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:31.673719 22770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:31.678985 22770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:31.680081 22770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:31.681028 22770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:31.689658 22775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:31.689666 22776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:31.689765 22770 server_base.cc:1034] running on GCE node
W20260620 19:09:31.689666 22778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:31.690450 22770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:31.691545 22770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:31.692715 22770 hybrid_clock.cc:648] HybridClock initialized: now 1781982571692699 us; error 30 us; skew 500 ppm
Jun 20 19:09:31 dist-test-slave-6wrf krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:31.695528 22770 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:09:31.696367 22770 webserver.cc:460] Webserver started at http://127.0.0.1:46033/ using document root <none> and password file <none>
I20260620 19:09:31.696873 22770 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:31.696936 22770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:31.697105 22770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:31.699265 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "8564252a262e4262abd519d2a8aba5c9"
format_stamp: "Formatted at 2026-06-20 19:09:31 on dist-test-slave-6wrf"
server_key: "197177e5b257a638d1edbb9e8491987f"
server_key_iv: "1cec049216cd59a32563be8b2f43b887"
server_key_version: "encryptionkey@0"
I20260620 19:09:31.700614 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "8564252a262e4262abd519d2a8aba5c9"
format_stamp: "Formatted at 2026-06-20 19:09:31 on dist-test-slave-6wrf"
server_key: "197177e5b257a638d1edbb9e8491987f"
server_key_iv: "1cec049216cd59a32563be8b2f43b887"
server_key_version: "encryptionkey@0"
I20260620 19:09:31.705144 22770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:09:31.707700 22785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:31.709012 22770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:09:31.709160 22770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "8564252a262e4262abd519d2a8aba5c9"
format_stamp: "Formatted at 2026-06-20 19:09:31 on dist-test-slave-6wrf"
server_key: "197177e5b257a638d1edbb9e8491987f"
server_key_iv: "1cec049216cd59a32563be8b2f43b887"
server_key_version: "encryptionkey@0"
I20260620 19:09:31.709280 22770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:31.713737 22770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:31.717054 22770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:31.717281 22770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:31.725183 22770 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46595
I20260620 19:09:31.725281 22837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46595 every 8 connection(s)
I20260620 19:09:31.726307 22770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:31.730356 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 22770
I20260620 19:09:31.730415 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:31.730665 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:31.731302 1642 external_mini_cluster.cc:1428] Setting key 335b5dcf987d8c12fbc791b4aebbb255
I20260620 19:09:31.736730 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:31.732948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:47123 (local address 127.0.0.1:46595)
0620 19:09:31.733354 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:31.733369 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:31.733403 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:09:31.734072 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:31.734088 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:31.734353 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:31.734636 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:31.735515 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:31.735521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:31.735531 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:31.735542 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:31.735597 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:31.735615 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:31.735737 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:31.735857 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 19:09:31.736030 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":133,"threads_started":1}
I20260620 19:09:31.738238 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9: Bootstrap starting.
I20260620 19:09:31.739441 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:31.740581 22838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:31.741406 22838 log.cc:824] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:31.743314 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:31.741532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:37103 (local address 127.0.0.1:46595)
0620 19:09:31.741694 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:31.741700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:31.741934 (+ 234us) server_negotiation.cc:407] Connection header received
0620 19:09:31.742070 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:31.742073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:31.742112 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:31.742202 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:31.742931 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:31.742935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:31.742937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:31.742949 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:31.742981 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:31.742990 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:31.743080 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:31.743153 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 19:09:31.743194 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:09:31.743467 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9: No bootstrap required, opened a new log
I20260620 19:09:31.747087 22838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8564252a262e4262abd519d2a8aba5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260620 19:09:31.747347 22838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:31.747387 22838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8564252a262e4262abd519d2a8aba5c9, State: Initialized, Role: FOLLOWER
I20260620 19:09:31.747851 22838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8564252a262e4262abd519d2a8aba5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260620 19:09:31.747973 22838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:31.748014 22838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:31.748122 22838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:31.748992 22838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8564252a262e4262abd519d2a8aba5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260620 19:09:31.749330 22838 leader_election.cc:302] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8564252a262e4262abd519d2a8aba5c9; no voters:
I20260620 19:09:31.749691 22838 leader_election.cc:288] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:31.749806 22848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:31.750058 22848 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [term 1 LEADER]: Becoming Leader. State: Replica: 8564252a262e4262abd519d2a8aba5c9, State: Running, Role: LEADER
I20260620 19:09:31.750442 22848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8564252a262e4262abd519d2a8aba5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260620 19:09:31.750859 22838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:31.752017 22849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8564252a262e4262abd519d2a8aba5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8564252a262e4262abd519d2a8aba5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } } }
I20260620 19:09:31.752142 22849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:31.752122 22850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8564252a262e4262abd519d2a8aba5c9. Latest consensus state: current_term: 1 leader_uuid: "8564252a262e4262abd519d2a8aba5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8564252a262e4262abd519d2a8aba5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } } }
I20260620 19:09:31.752499 22850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:31.752516 22857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:31.755594 22857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:31.761021 22857 catalog_manager.cc:1347] Generated new cluster ID: 8e2fb91546114d93b50b3ed5875c281f
I20260620 19:09:31.761116 22857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:31.772300 22857 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:31.772670 22857 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:31.786072 22857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8564252a262e4262abd519d2a8aba5c9: Generated new TSK 0
I20260620 19:09:31.786923 22857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:31.798928 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 22770
2026-06-20T19:09:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:31 dist-test-slave-6wrf krb5kdc[22869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:31 dist-test-slave-6wrf krb5kdc[22869](info): set up 2 sockets
Jun 20 19:09:31 dist-test-slave-6wrf krb5kdc[22869](info): commencing operation
krb5kdc: starting...
W20260620 19:09:32.857882 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:32 dist-test-slave-6wrf krb5kdc[22869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:09:32.983484 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37205
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:33.095760 22882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:33.096036 22882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:33.096078 22882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:33.100013 22882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:33.100085 22882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:33.100106 22882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:33.100121 22882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:33.100136 22882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:33.105352 22882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37205
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:33.106457 22882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:33.107478 22882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:33.116920 22887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:33.116914 22890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:33.116888 22888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:33.117303 22882 server_base.cc:1034] running on GCE node
I20260620 19:09:33.118050 22882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:33.119408 22882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:33.120730 22882 hybrid_clock.cc:648] HybridClock initialized: now 1781982573120693 us; error 37 us; skew 500 ppm
Jun 20 19:09:33 dist-test-slave-6wrf krb5kdc[22869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:33.124050 22882 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:09:33.125026 22882 webserver.cc:460] Webserver started at http://127.0.0.1:42663/ using document root <none> and password file <none>
I20260620 19:09:33.125677 22882 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:33.125767 22882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:33.125986 22882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:33.128087 22882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "854a9320b4424bd983e1b6319531d2fa"
format_stamp: "Formatted at 2026-06-20 19:09:33 on dist-test-slave-6wrf"
server_key: "16bbb7fa24abd435d86062595b14a6ba"
server_key_iv: "b3fff06b4a453e21fa307aa8d88ccb34"
server_key_version: "encryptionkey@0"
I20260620 19:09:33.128679 22882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "854a9320b4424bd983e1b6319531d2fa"
format_stamp: "Formatted at 2026-06-20 19:09:33 on dist-test-slave-6wrf"
server_key: "16bbb7fa24abd435d86062595b14a6ba"
server_key_iv: "b3fff06b4a453e21fa307aa8d88ccb34"
server_key_version: "encryptionkey@0"
I20260620 19:09:33.132856 22882 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:09:33.135731 22897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:33.137036 22882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 19:09:33.137184 22882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "854a9320b4424bd983e1b6319531d2fa"
format_stamp: "Formatted at 2026-06-20 19:09:33 on dist-test-slave-6wrf"
server_key: "16bbb7fa24abd435d86062595b14a6ba"
server_key_iv: "b3fff06b4a453e21fa307aa8d88ccb34"
server_key_version: "encryptionkey@0"
I20260620 19:09:33.137307 22882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:33.152865 22882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:33.156111 22882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:33.156323 22882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:33.163573 22882 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37205
I20260620 19:09:33.163867 22949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37205 every 8 connection(s)
I20260620 19:09:33.164516 22882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:33.167245 22950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:33.172133 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 22882
I20260620 19:09:33.172358 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:33.172701 1642 external_mini_cluster.cc:1428] Setting key 3c919dd00e81fe1ff24a4873713e8c90
I20260620 19:09:33.172958 22950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa: Bootstrap starting.
I20260620 19:09:33.175997 22950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:33.176975 22950 log.cc:824] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:33.179186 22950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa: No bootstrap required, opened a new log
Jun 20 19:09:33 dist-test-slave-6wrf krb5kdc[22869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:09:33.183619 22950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854a9320b4424bd983e1b6319531d2fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37205 } }
I20260620 19:09:33.183894 22950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:33.183944 22950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854a9320b4424bd983e1b6319531d2fa, State: Initialized, Role: FOLLOWER
I20260620 19:09:33.184433 22950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854a9320b4424bd983e1b6319531d2fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37205 } }
I20260620 19:09:33.184617 22950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:33.184688 22950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:33.184800 22950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:33.186066 22950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854a9320b4424bd983e1b6319531d2fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37205 } }
I20260620 19:09:33.186437 22950 leader_election.cc:302] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 854a9320b4424bd983e1b6319531d2fa; no voters:
I20260620 19:09:33.186806 22950 leader_election.cc:288] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:33.186892 22955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:33.187076 22955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [term 1 LEADER]: Becoming Leader. State: Replica: 854a9320b4424bd983e1b6319531d2fa, State: Running, Role: LEADER
I20260620 19:09:33.187404 22955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854a9320b4424bd983e1b6319531d2fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37205 } }
I20260620 19:09:33.188148 22950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:33.188642 22956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "854a9320b4424bd983e1b6319531d2fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854a9320b4424bd983e1b6319531d2fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37205 } } }
I20260620 19:09:33.188766 22956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:33.189443 22957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 854a9320b4424bd983e1b6319531d2fa. Latest consensus state: current_term: 1 leader_uuid: "854a9320b4424bd983e1b6319531d2fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854a9320b4424bd983e1b6319531d2fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37205 } } }
I20260620 19:09:33.189548 22957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:33.189860 22960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:33.191380 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:33.174495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:37401 (local address 127.0.0.1:37205)
0620 19:09:33.174980 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:33.174995 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:33.175033 (+ 38us) server_negotiation.cc:407] Connection header received
0620 19:09:33.175657 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:33.175672 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:33.176003 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:33.176346 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:33.177295 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:33.178553 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:33.179724 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:33.180203 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:33.180325 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:33.183948 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:33.183957 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:33.183975 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:33.184011 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:33.186116 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:33.187305 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:33.187316 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:33.187326 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:33.187576 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:33.188057 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:33.188063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:33.188068 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:33.188864 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:33.189105 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:33.190175 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0620 19:09:33.190410 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":165,"threads_started":1}
I20260620 19:09:33.194278 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:33.194424 22960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:33.200155 22960 catalog_manager.cc:1347] Generated new cluster ID: c20a95faf67f457ebb0d4a58b06329a1
I20260620 19:09:33.200222 22960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:33.205935 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:33.196530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:43145 (local address 127.0.0.1:37205)
0620 19:09:33.196630 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:33.196634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:33.197143 (+ 509us) server_negotiation.cc:407] Connection header received
0620 19:09:33.197253 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:33.197256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:33.197303 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:33.197384 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:33.198352 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:33.199014 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:33.200168 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:33.200518 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:33.200596 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:33.202533 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:33.202536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:33.202539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:33.202562 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:33.204140 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:33.204664 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:33.204667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:33.204669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:33.204743 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:33.205081 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:33.205084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:33.205086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:33.205268 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:33.205399 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:33.205700 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 19:09:33.205785 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:09:33.225929 22960 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:33.227101 22960 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:33.236449 22960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 854a9320b4424bd983e1b6319531d2fa: Generated new TSK 0
I20260620 19:09:33.237267 22960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:33.243429 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 22882
W20260620 19:09:33.249084 22867 connection.cc:535] client connection to 127.0.0.1:37205 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37205): Connection reset by peer (error 104)
2026-06-20T19:09:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:33 dist-test-slave-6wrf krb5kdc[22981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:33 dist-test-slave-6wrf krb5kdc[22981](info): set up 2 sockets
Jun 20 19:09:33 dist-test-slave-6wrf krb5kdc[22981](info): commencing operation
krb5kdc: starting...
W20260620 19:09:34.314977 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:34 dist-test-slave-6wrf krb5kdc[22981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:09:34.442034 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:34.561404 22994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:34.561718 22994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:34.561808 22994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:34.565917 22994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:34.565987 22994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:34.566010 22994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:34.566030 22994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:34.566092 22994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:34.571585 22994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34513
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:34.572722 22994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:34.573797 22994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:34.582624 23002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:34.582743 22994 server_base.cc:1034] running on GCE node
W20260620 19:09:34.582592 23000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:34.582576 22999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:34.583424 22994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:34.584514 22994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:34.585693 22994 hybrid_clock.cc:648] HybridClock initialized: now 1781982574585644 us; error 58 us; skew 500 ppm
Jun 20 19:09:34 dist-test-slave-6wrf krb5kdc[22981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:34.589247 22994 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:09:34.590304 22994 webserver.cc:460] Webserver started at http://127.0.0.1:40061/ using document root <none> and password file <none>
I20260620 19:09:34.590909 22994 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:34.591006 22994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:34.591239 22994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:34.593374 22994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "1ccec3884ce2450e98222193ad3ed52d"
format_stamp: "Formatted at 2026-06-20 19:09:34 on dist-test-slave-6wrf"
server_key: "fcc65305525b72d435a3fcf6358205f6"
server_key_iv: "c4c259dbd4fc06e1e9cdaaf6a0448cb2"
server_key_version: "encryptionkey@0"
I20260620 19:09:34.594002 22994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "1ccec3884ce2450e98222193ad3ed52d"
format_stamp: "Formatted at 2026-06-20 19:09:34 on dist-test-slave-6wrf"
server_key: "fcc65305525b72d435a3fcf6358205f6"
server_key_iv: "c4c259dbd4fc06e1e9cdaaf6a0448cb2"
server_key_version: "encryptionkey@0"
I20260620 19:09:34.598138 22994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:09:34.601027 23009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:34.603009 22994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:34.603250 22994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "1ccec3884ce2450e98222193ad3ed52d"
format_stamp: "Formatted at 2026-06-20 19:09:34 on dist-test-slave-6wrf"
server_key: "fcc65305525b72d435a3fcf6358205f6"
server_key_iv: "c4c259dbd4fc06e1e9cdaaf6a0448cb2"
server_key_version: "encryptionkey@0"
I20260620 19:09:34.603458 22994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:34.626426 22994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:34.629890 22994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:34.630116 22994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:34.638141 22994 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34513
I20260620 19:09:34.638160 23061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34513 every 8 connection(s)
I20260620 19:09:34.639209 22994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:34.641965 23062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:34.649367 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 22994
I20260620 19:09:34.649520 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:34.649415 23062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d: Bootstrap starting.
I20260620 19:09:34.649973 1642 external_mini_cluster.cc:1428] Setting key d6ec792f787158fe1f89d6dc1fa82fdc
I20260620 19:09:34.652987 23062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:34.653826 23062 log.cc:824] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:34.656639 23062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d: No bootstrap required, opened a new log
I20260620 19:09:34.660184 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:34.651880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:49807 (local address 127.0.0.1:34513)
0620 19:09:34.652437 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:34.652454 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:34.652495 (+ 41us) server_negotiation.cc:407] Connection header received
0620 19:09:34.653348 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:34.653367 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:34.653804 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:34.654172 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:34.654617 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:34.655595 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:34.656484 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:34.657016 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:34.657204 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:34.658242 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:34.658248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:34.658270 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:34.658668 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:34.658720 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:34.658743 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:34.658888 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:34.659073 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 19:09:34.659212 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":159,"threads_started":1}
I20260620 19:09:34.661929 23062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ccec3884ce2450e98222193ad3ed52d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34513 } }
I20260620 19:09:34.662405 23062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:34.662510 23062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ccec3884ce2450e98222193ad3ed52d, State: Initialized, Role: FOLLOWER
I20260620 19:09:34.663249 23062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ccec3884ce2450e98222193ad3ed52d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34513 } }
I20260620 19:09:34.663455 23062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:34.663533 23062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:34.663669 23062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:34.664225 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:34.665181 23062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ccec3884ce2450e98222193ad3ed52d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34513 } }
I20260620 19:09:34.665920 23062 leader_election.cc:302] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ccec3884ce2450e98222193ad3ed52d; no voters:
I20260620 19:09:34.666275 23062 leader_election.cc:288] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:34.666359 23071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:34.666536 23071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [term 1 LEADER]: Becoming Leader. State: Replica: 1ccec3884ce2450e98222193ad3ed52d, State: Running, Role: LEADER
I20260620 19:09:34.666858 23071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ccec3884ce2450e98222193ad3ed52d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34513 } }
I20260620 19:09:34.667641 23062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:34.668144 23071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ccec3884ce2450e98222193ad3ed52d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ccec3884ce2450e98222193ad3ed52d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34513 } } }
I20260620 19:09:34.668268 23071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:34.668470 23071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ccec3884ce2450e98222193ad3ed52d. Latest consensus state: current_term: 1 leader_uuid: "1ccec3884ce2450e98222193ad3ed52d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ccec3884ce2450e98222193ad3ed52d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34513 } } }
I20260620 19:09:34.668577 23071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:34.671551 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:34.666708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:36631 (local address 127.0.0.1:34513)
0620 19:09:34.666898 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:34.666902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:34.666915 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:09:34.666954 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:34.666956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:34.667043 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:34.667116 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:34.667482 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:34.668139 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:34.669292 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:34.669633 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:34.669729 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:34.670711 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:34.670715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:34.670718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:34.670731 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:34.670759 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:34.670771 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:34.670939 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:34.671376 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 19:09:34.671416 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
W20260620 19:09:34.673084 23087 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:09:34.673182 23087 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:09:34.673265 23077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:34.674101 23077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:34.680814 23077 catalog_manager.cc:1347] Generated new cluster ID: 56e0fafc82fb451ab0638437d84d8f0b
I20260620 19:09:34.680900 23077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:34.697216 23077 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:34.698378 23077 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:34.704504 23077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ccec3884ce2450e98222193ad3ed52d: Generated new TSK 0
I20260620 19:09:34.705268 23077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:34.711189 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 22994
W20260620 19:09:34.716245 22979 connection.cc:535] client connection to 127.0.0.1:34513 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34513): Connection reset by peer (error 104)
2026-06-20T19:09:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:34 dist-test-slave-6wrf krb5kdc[23093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:34 dist-test-slave-6wrf krb5kdc[23093](info): set up 2 sockets
Jun 20 19:09:34 dist-test-slave-6wrf krb5kdc[23093](info): commencing operation
krb5kdc: starting...
W20260620 19:09:35.790027 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:35 dist-test-slave-6wrf krb5kdc[23093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:09:35.922469 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:36.037175 23106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:36.037463 23106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:36.037516 23106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:36.042328 23106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:36.042448 23106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:36.042472 23106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:36.042493 23106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:36.042513 23106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:36.049440 23106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39501
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:36.050631 23106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:36.051625 23106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:36.061744 23112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:36.061810 23111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:36.061744 23114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:36.062456 23106 server_base.cc:1034] running on GCE node
I20260620 19:09:36.063048 23106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:36.064101 23106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:36.065352 23106 hybrid_clock.cc:648] HybridClock initialized: now 1781982576065315 us; error 46 us; skew 500 ppm
Jun 20 19:09:36 dist-test-slave-6wrf krb5kdc[23093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:36.074276 23106 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:09:36.075309 23106 webserver.cc:460] Webserver started at http://127.0.0.1:40457/ using document root <none> and password file <none>
I20260620 19:09:36.075902 23106 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:36.075994 23106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:36.076217 23106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:36.078426 23106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "7d9c9ab149ba4b3fb2e41d9f248175dd"
format_stamp: "Formatted at 2026-06-20 19:09:36 on dist-test-slave-6wrf"
server_key: "50f4ae75c223cb3f1ddb8c432692700f"
server_key_iv: "19b1e701f3fccec61accdea810e57943"
server_key_version: "encryptionkey@0"
I20260620 19:09:36.079007 23106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "7d9c9ab149ba4b3fb2e41d9f248175dd"
format_stamp: "Formatted at 2026-06-20 19:09:36 on dist-test-slave-6wrf"
server_key: "50f4ae75c223cb3f1ddb8c432692700f"
server_key_iv: "19b1e701f3fccec61accdea810e57943"
server_key_version: "encryptionkey@0"
I20260620 19:09:36.082896 23106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:09:36.085479 23121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:36.086863 23106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:36.087028 23106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "7d9c9ab149ba4b3fb2e41d9f248175dd"
format_stamp: "Formatted at 2026-06-20 19:09:36 on dist-test-slave-6wrf"
server_key: "50f4ae75c223cb3f1ddb8c432692700f"
server_key_iv: "19b1e701f3fccec61accdea810e57943"
server_key_version: "encryptionkey@0"
I20260620 19:09:36.087177 23106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:36.091832 23106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:36.095568 23106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:36.095782 23106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:36.103006 23106 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39501
I20260620 19:09:36.103024 23173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39501 every 8 connection(s)
I20260620 19:09:36.103986 23106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:36.106808 23174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:36.109586 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 23106
I20260620 19:09:36.109767 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:36.110072 1642 external_mini_cluster.cc:1428] Setting key 7ade845fe809e11537f1a6690cb85a25
I20260620 19:09:36.112787 23174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd: Bootstrap starting.
I20260620 19:09:36.115123 23174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:36.115952 23174 log.cc:824] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:36.116153 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:36.111890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:48145 (local address 127.0.0.1:39501)
0620 19:09:36.112612 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:36.112627 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:36.112666 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:09:36.113368 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:36.113386 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:36.113766 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:36.114118 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:36.114953 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:36.114958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:36.114969 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:36.114979 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:36.115029 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:36.115048 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:36.115165 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:36.115262 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 19:09:36.115429 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":167,"threads_started":1}
I20260620 19:09:36.118525 23174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd: No bootstrap required, opened a new log
I20260620 19:09:36.119822 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:36.123057 23174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9c9ab149ba4b3fb2e41d9f248175dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39501 } }
I20260620 19:09:36.123371 23174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:36.123430 23174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9c9ab149ba4b3fb2e41d9f248175dd, State: Initialized, Role: FOLLOWER
I20260620 19:09:36.123776 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:36.122092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:52231 (local address 127.0.0.1:39501)
0620 19:09:36.122223 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:36.122228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:36.122369 (+ 141us) server_negotiation.cc:407] Connection header received
0620 19:09:36.122491 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:36.122493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:36.122523 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:36.122611 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:36.123321 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:36.123324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:36.123326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:36.123336 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:36.123359 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:36.123369 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:36.123476 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:36.123596 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 19:09:36.123639 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:09:36.123955 23174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9c9ab149ba4b3fb2e41d9f248175dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39501 } }
I20260620 19:09:36.124154 23174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:36.124224 23174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:36.124329 23174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:36.125459 23174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9c9ab149ba4b3fb2e41d9f248175dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39501 } }
I20260620 19:09:36.125835 23174 leader_election.cc:302] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d9c9ab149ba4b3fb2e41d9f248175dd; no voters:
I20260620 19:09:36.126101 23174 leader_election.cc:288] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:36.126495 23185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:36.126874 23185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [term 1 LEADER]: Becoming Leader. State: Replica: 7d9c9ab149ba4b3fb2e41d9f248175dd, State: Running, Role: LEADER
I20260620 19:09:36.127297 23185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9c9ab149ba4b3fb2e41d9f248175dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39501 } }
I20260620 19:09:36.127378 23174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:36.129587 23186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d9c9ab149ba4b3fb2e41d9f248175dd. Latest consensus state: current_term: 1 leader_uuid: "7d9c9ab149ba4b3fb2e41d9f248175dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9c9ab149ba4b3fb2e41d9f248175dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39501 } } }
I20260620 19:09:36.129979 23186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:36.129774 23184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d9c9ab149ba4b3fb2e41d9f248175dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9c9ab149ba4b3fb2e41d9f248175dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39501 } } }
I20260620 19:09:36.130211 23184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd [sys.catalog]: This master's current role is: LEADER
W20260620 19:09:36.132463 23198 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:09:36.132536 23198 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:09:36.132598 23199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:36.133332 23199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:36.138967 23199 catalog_manager.cc:1347] Generated new cluster ID: 4a04476d4ee74883ba998c0d3ca4024c
I20260620 19:09:36.139041 23199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:36.163005 23199 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:36.163446 23199 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:36.190253 23199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d9c9ab149ba4b3fb2e41d9f248175dd: Generated new TSK 0
I20260620 19:09:36.191180 23199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:36.196775 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 23106
2026-06-20T19:09:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1485 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:36 dist-test-slave-6wrf krb5kdc[23205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:36 dist-test-slave-6wrf krb5kdc[23205](info): set up 2 sockets
Jun 20 19:09:36 dist-test-slave-6wrf krb5kdc[23205](info): commencing operation
krb5kdc: starting...
W20260620 19:09:37.281972 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:37 dist-test-slave-6wrf krb5kdc[23205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:09:37.413589 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43435
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:46567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:43435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:37.536782 23218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:37.537065 23218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:37.537110 23218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:37.541340 23218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:37.541409 23218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:37.541428 23218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:37.541445 23218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:37.541461 23218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:37.546872 23218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:46567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:43435
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:37.547971 23218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:37.549010 23218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:37.557749 23226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:37.557876 23223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:37.557765 23224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:37.558112 23218 server_base.cc:1034] running on GCE node
I20260620 19:09:37.558658 23218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:37.559803 23218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:37.560981 23218 hybrid_clock.cc:648] HybridClock initialized: now 1781982577560930 us; error 67 us; skew 500 ppm
Jun 20 19:09:37 dist-test-slave-6wrf krb5kdc[23205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:37.564363 23218 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:09:37.565282 23218 webserver.cc:460] Webserver started at http://127.1.154.190:34651/ using document root <none> and password file <none>
I20260620 19:09:37.565928 23218 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:37.566025 23218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:37.566277 23218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:37.568499 23218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "8eb8af377eaa4ca5b24cbdcb8322a003"
format_stamp: "Formatted at 2026-06-20 19:09:37 on dist-test-slave-6wrf"
server_key: "f0267c1f5b78ee98aadc3081b3a106c3"
server_key_iv: "ef102897abd6661e23f2a85650ef4d4a"
server_key_version: "encryptionkey@0"
I20260620 19:09:37.569099 23218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "8eb8af377eaa4ca5b24cbdcb8322a003"
format_stamp: "Formatted at 2026-06-20 19:09:37 on dist-test-slave-6wrf"
server_key: "f0267c1f5b78ee98aadc3081b3a106c3"
server_key_iv: "ef102897abd6661e23f2a85650ef4d4a"
server_key_version: "encryptionkey@0"
I20260620 19:09:37.573542 23218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:09:37.577124 23233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:37.578730 23218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:37.578919 23218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "8eb8af377eaa4ca5b24cbdcb8322a003"
format_stamp: "Formatted at 2026-06-20 19:09:37 on dist-test-slave-6wrf"
server_key: "f0267c1f5b78ee98aadc3081b3a106c3"
server_key_iv: "ef102897abd6661e23f2a85650ef4d4a"
server_key_version: "encryptionkey@0"
I20260620 19:09:37.579064 23218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:37.607029 23218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:37.612099 23218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:37.612416 23218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:37.620774 23218 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:43435
I20260620 19:09:37.620770 23285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:43435 every 8 connection(s)
I20260620 19:09:37.621803 23218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:37.624652 23286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:37.630450 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003: Bootstrap starting.
I20260620 19:09:37.631243 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 23218
I20260620 19:09:37.631477 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:37.631812 1642 external_mini_cluster.cc:1428] Setting key da0c56357152c4b280f61aab998b2ce9
I20260620 19:09:37.633303 23286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:37.634217 23286 log.cc:824] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:37.636889 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003: No bootstrap required, opened a new log
I20260620 19:09:37.641377 23286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb8af377eaa4ca5b24cbdcb8322a003" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43435 } }
I20260620 19:09:37.641721 23286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:37.641781 23286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb8af377eaa4ca5b24cbdcb8322a003, State: Initialized, Role: FOLLOWER
I20260620 19:09:37.642342 23286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb8af377eaa4ca5b24cbdcb8322a003" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43435 } }
I20260620 19:09:37.642529 23286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:37.642594 23286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:37.642690 23286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 19:09:37 dist-test-slave-6wrf krb5kdc[23205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:37.643831 23286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb8af377eaa4ca5b24cbdcb8322a003" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43435 } }
I20260620 19:09:37.644205 23286 leader_election.cc:302] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eb8af377eaa4ca5b24cbdcb8322a003; no voters:
I20260620 19:09:37.644636 23286 leader_election.cc:288] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:37.644786 23291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:37.645076 23291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [term 1 LEADER]: Becoming Leader. State: Replica: 8eb8af377eaa4ca5b24cbdcb8322a003, State: Running, Role: LEADER
I20260620 19:09:37.645447 23291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb8af377eaa4ca5b24cbdcb8322a003" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43435 } }
I20260620 19:09:37.646119 23286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:37.647284 23292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eb8af377eaa4ca5b24cbdcb8322a003. Latest consensus state: current_term: 1 leader_uuid: "8eb8af377eaa4ca5b24cbdcb8322a003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb8af377eaa4ca5b24cbdcb8322a003" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43435 } } }
I20260620 19:09:37.647696 23292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:37.647579 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8eb8af377eaa4ca5b24cbdcb8322a003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb8af377eaa4ca5b24cbdcb8322a003" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43435 } } }
I20260620 19:09:37.648012 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:37.650326 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:37.633431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38411 (local address 127.1.154.190:43435)
0620 19:09:37.634119 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:37.634139 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:37.634188 (+ 49us) server_negotiation.cc:407] Connection header received
0620 19:09:37.635328 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:37.635358 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:37.635835 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:37.636238 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:37.637389 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:37.638623 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:37.639898 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:37.640440 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:37.640584 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:37.643859 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:37.643866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:37.643884 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:37.643919 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:37.646129 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:37.646869 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:37.646875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:37.646883 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:37.647053 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:37.647656 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:37.647659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:37.647662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:37.648230 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:37.648444 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:37.649028 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 19:09:37.649285 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":183,"threads_started":1}
W20260620 19:09:37.651710 23306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:09:37.651813 23306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:09:37.651914 23303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:37.652837 23303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:37.653672 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:37.660043 23303 catalog_manager.cc:1347] Generated new cluster ID: 605ed0115b3b499488d5363bf55c851f
I20260620 19:09:37.660122 23303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:37.668609 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:37.656101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55739 (local address 127.1.154.190:43435)
0620 19:09:37.656263 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:37.656268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:37.656534 (+ 266us) server_negotiation.cc:407] Connection header received
0620 19:09:37.656986 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:37.657043 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:37.657148 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:37.657275 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:37.658745 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:37.659592 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:37.662611 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:37.663040 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:37.663177 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:37.664822 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:37.664827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:37.664830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:37.664860 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:37.666678 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:37.667204 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:37.667208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:37.667210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:37.667319 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:37.667662 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:37.667666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:37.667668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:37.667899 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:37.668055 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:37.668317 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 19:09:37.668425 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 19:09:37.689797 23303 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:37.691465 23303 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:37.704880 23303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8eb8af377eaa4ca5b24cbdcb8322a003: Generated new TSK 0
I20260620 19:09:37.705716 23303 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:37.711184 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 23218
W20260620 19:09:37.716176 23203 connection.cc:535] client connection to 127.1.154.190:43435 recv error: Network error: failed to read from TLS socket (remote: 127.1.154.190:43435): Connection reset by peer (error 104)
2026-06-20T19:09:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1513 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:37 dist-test-slave-6wrf krb5kdc[23317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:37 dist-test-slave-6wrf krb5kdc[23317](info): set up 2 sockets
Jun 20 19:09:37 dist-test-slave-6wrf krb5kdc[23317](info): commencing operation
krb5kdc: starting...
W20260620 19:09:38.779294 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:38 dist-test-slave-6wrf krb5kdc[23317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:09:38.914003 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:44213
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:44213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:39.032411 23330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:39.032747 23330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:39.032835 23330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:39.037041 23330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:39.037146 23330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:39.037187 23330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:39.037249 23330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:39.037288 23330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:39.042721 23330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:44213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:44213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:39.043854 23330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:39.044896 23330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:39.055426 23336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:39.055630 23335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:39.055670 23338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:39.055791 23330 server_base.cc:1034] running on GCE node
I20260620 19:09:39.056542 23330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:39.057818 23330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:39.059106 23330 hybrid_clock.cc:648] HybridClock initialized: now 1781982579059072 us; error 39 us; skew 500 ppm
Jun 20 19:09:39 dist-test-slave-6wrf krb5kdc[23317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982579, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:39.062857 23330 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:09:39.063886 23330 webserver.cc:460] Webserver started at http://127.1.154.190:44607/ using document root <none> and password file <none>
I20260620 19:09:39.064504 23330 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:39.064604 23330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:39.064862 23330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:39.067129 23330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "c0a12b2617fd430e9483eec48d775775"
format_stamp: "Formatted at 2026-06-20 19:09:39 on dist-test-slave-6wrf"
server_key: "4b8114cddec0efdb109b272af5f27564"
server_key_iv: "866b13457a852260e492f608e196c94d"
server_key_version: "encryptionkey@0"
I20260620 19:09:39.067741 23330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "c0a12b2617fd430e9483eec48d775775"
format_stamp: "Formatted at 2026-06-20 19:09:39 on dist-test-slave-6wrf"
server_key: "4b8114cddec0efdb109b272af5f27564"
server_key_iv: "866b13457a852260e492f608e196c94d"
server_key_version: "encryptionkey@0"
I20260620 19:09:39.071919 23330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:09:39.074494 23345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:39.076480 23330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:39.076748 23330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "c0a12b2617fd430e9483eec48d775775"
format_stamp: "Formatted at 2026-06-20 19:09:39 on dist-test-slave-6wrf"
server_key: "4b8114cddec0efdb109b272af5f27564"
server_key_iv: "866b13457a852260e492f608e196c94d"
server_key_version: "encryptionkey@0"
I20260620 19:09:39.076926 23330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:39.100523 23330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:39.104043 23330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:39.104281 23330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:39.112596 23330 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:44213
I20260620 19:09:39.112682 23397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:44213 every 8 connection(s)
I20260620 19:09:39.113691 23330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:39.116427 23398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:39.121902 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 23330
I20260620 19:09:39.121977 23398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775: Bootstrap starting.
I20260620 19:09:39.122057 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:39.122344 1642 external_mini_cluster.cc:1428] Setting key 61ab3ee7f4eac5f13ab10d00dfd85f4e
I20260620 19:09:39.124531 23398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:39.125242 23398 log.cc:824] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:39.127224 23398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775: No bootstrap required, opened a new log
I20260620 19:09:39.131413 23398 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a12b2617fd430e9483eec48d775775" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 44213 } }
I20260620 19:09:39.131662 23398 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:39.131739 23398 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0a12b2617fd430e9483eec48d775775, State: Initialized, Role: FOLLOWER
I20260620 19:09:39.131901 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:39.123748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56109 (local address 127.1.154.190:44213)
0620 19:09:39.124154 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:39.124169 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:39.124203 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:09:39.124906 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:39.124923 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:39.125253 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:39.125573 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:39.126075 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:39.127020 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:39.128199 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:39.128677 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:39.128803 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:39.129867 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:39.129872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:39.129884 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:39.130420 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:39.130483 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:39.130508 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:39.130698 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:39.130925 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 19:09:39.131057 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":132,"threads_started":1}
I20260620 19:09:39.132242 23398 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a12b2617fd430e9483eec48d775775" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 44213 } }
I20260620 19:09:39.132398 23398 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:39.132467 23398 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:39.132565 23398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:39.133721 23398 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a12b2617fd430e9483eec48d775775" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 44213 } }
I20260620 19:09:39.134074 23398 leader_election.cc:302] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0a12b2617fd430e9483eec48d775775; no voters:
I20260620 19:09:39.134438 23398 leader_election.cc:288] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:39.134497 23403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:39.134696 23403 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [term 1 LEADER]: Becoming Leader. State: Replica: c0a12b2617fd430e9483eec48d775775, State: Running, Role: LEADER
I20260620 19:09:39.134984 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:39.135041 23403 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a12b2617fd430e9483eec48d775775" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 44213 } }
I20260620 19:09:39.135689 23398 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:39.136670 23405 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0a12b2617fd430e9483eec48d775775. Latest consensus state: current_term: 1 leader_uuid: "c0a12b2617fd430e9483eec48d775775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a12b2617fd430e9483eec48d775775" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 44213 } } }
I20260620 19:09:39.136812 23405 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:39.137198 23404 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0a12b2617fd430e9483eec48d775775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a12b2617fd430e9483eec48d775775" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 44213 } } }
I20260620 19:09:39.137305 23404 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:39.137848 23416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:39.141213 23416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:39.142448 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:39.137075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55263 (local address 127.1.154.190:44213)
0620 19:09:39.137172 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:39.137178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:39.138191 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 19:09:39.138401 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:39.138404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:39.138449 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:39.138533 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:39.138877 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:39.139582 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:39.140456 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:39.140709 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:39.140818 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:39.141801 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:39.141804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:39.141806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:39.141822 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:39.141846 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:39.141857 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:39.141956 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:39.142231 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 19:09:39.142274 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:09:39.146883 23416 catalog_manager.cc:1347] Generated new cluster ID: 78b6fe1fac0f4827b9c3e184aa2e5bb5
I20260620 19:09:39.146951 23416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:39.160781 23416 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:39.162051 23416 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:39.177321 23416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0a12b2617fd430e9483eec48d775775: Generated new TSK 0
I20260620 19:09:39.178309 23416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:39.194624 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 23330
W20260620 19:09:39.199947 23315 connection.cc:535] client connection to 127.1.154.190:44213 recv error: Network error: failed to read from TLS socket (remote: 127.1.154.190:44213): Connection reset by peer (error 104)
2026-06-20T19:09:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:39 dist-test-slave-6wrf krb5kdc[23429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:39 dist-test-slave-6wrf krb5kdc[23429](info): set up 2 sockets
Jun 20 19:09:39 dist-test-slave-6wrf krb5kdc[23429](info): commencing operation
krb5kdc: starting...
W20260620 19:09:40.278116 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:40 dist-test-slave-6wrf krb5kdc[23429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:09:40.409572 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:38095
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:46803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:38095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:40.533252 23442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:40.533689 23442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:40.533777 23442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:40.539528 23442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:40.539609 23442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:40.539633 23442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:40.539654 23442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:40.539673 23442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:40.545432 23442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:46803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:38095
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:38095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:40.546669 23442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:40.547698 23442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:40.559288 23447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:40.559288 23450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:40.559521 23448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:40.559824 23442 server_base.cc:1034] running on GCE node
I20260620 19:09:40.560418 23442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:40.561673 23442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:40.562951 23442 hybrid_clock.cc:648] HybridClock initialized: now 1781982580562922 us; error 47 us; skew 500 ppm
Jun 20 19:09:40 dist-test-slave-6wrf krb5kdc[23429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982580, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:40.566442 23442 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:09:40.567389 23442 webserver.cc:460] Webserver started at http://127.1.154.190:42865/ using document root <none> and password file <none>
I20260620 19:09:40.568001 23442 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:40.568095 23442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:40.568701 23442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:40.571512 23442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "f46ac12998d3434abe3581d2dd12f080"
format_stamp: "Formatted at 2026-06-20 19:09:40 on dist-test-slave-6wrf"
server_key: "ddbf2a10ca025b0c09c6fa57f5ca19a1"
server_key_iv: "3a621990477c2c1f44ed9e263680b2d7"
server_key_version: "encryptionkey@0"
I20260620 19:09:40.572228 23442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "f46ac12998d3434abe3581d2dd12f080"
format_stamp: "Formatted at 2026-06-20 19:09:40 on dist-test-slave-6wrf"
server_key: "ddbf2a10ca025b0c09c6fa57f5ca19a1"
server_key_iv: "3a621990477c2c1f44ed9e263680b2d7"
server_key_version: "encryptionkey@0"
I20260620 19:09:40.577338 23442 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 19:09:40.580533 23457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:40.582023 23442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:40.582135 23442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "f46ac12998d3434abe3581d2dd12f080"
format_stamp: "Formatted at 2026-06-20 19:09:40 on dist-test-slave-6wrf"
server_key: "ddbf2a10ca025b0c09c6fa57f5ca19a1"
server_key_iv: "3a621990477c2c1f44ed9e263680b2d7"
server_key_version: "encryptionkey@0"
I20260620 19:09:40.582240 23442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:40.587029 23442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:40.591602 23442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:40.591764 23442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:40.601339 23442 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:38095
I20260620 19:09:40.601516 23509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:38095 every 8 connection(s)
I20260620 19:09:40.602550 23442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:40.606442 23510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:40.607339 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 23442
I20260620 19:09:40.607457 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:40.607797 1642 external_mini_cluster.cc:1428] Setting key f795003ae028712623ecd07ddfe0338b
I20260620 19:09:40.614598 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:40.609509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58719 (local address 127.1.154.190:38095)
0620 19:09:40.610177 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:40.610194 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:40.610244 (+ 50us) server_negotiation.cc:407] Connection header received
0620 19:09:40.611276 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:40.611299 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:40.611637 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:40.612027 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:40.612925 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:40.612930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:40.612948 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:40.612958 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:40.613013 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:40.613032 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:40.613202 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:40.613277 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 19:09:40.613441 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":235,"threads_started":1}
I20260620 19:09:40.616767 23510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080: Bootstrap starting.
I20260620 19:09:40.618623 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:40.619727 23510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:40.620713 23510 log.cc:824] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:40.623420 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:40.620981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36443 (local address 127.1.154.190:38095)
0620 19:09:40.621161 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:40.621169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:40.621520 (+ 351us) server_negotiation.cc:407] Connection header received
0620 19:09:40.621763 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:40.621766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:40.621825 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:40.621930 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:40.622941 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:40.622944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:40.622947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:40.622958 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:40.622988 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:40.623005 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:40.623126 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:40.623258 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 19:09:40.623298 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 19:09:40.623848 23510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080: No bootstrap required, opened a new log
I20260620 19:09:40.628902 23510 raft_consensus.cc:348] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46ac12998d3434abe3581d2dd12f080" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38095 } }
I20260620 19:09:40.629442 23510 raft_consensus.cc:374] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:40.629554 23510 raft_consensus.cc:729] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46ac12998d3434abe3581d2dd12f080, State: Initialized, Role: FOLLOWER
I20260620 19:09:40.630448 23510 consensus_queue.cc:260] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46ac12998d3434abe3581d2dd12f080" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38095 } }
I20260620 19:09:40.630716 23510 raft_consensus.cc:388] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:40.630786 23510 raft_consensus.cc:482] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:40.630945 23510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:40.632472 23510 raft_consensus.cc:504] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46ac12998d3434abe3581d2dd12f080" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38095 } }
I20260620 19:09:40.633038 23510 leader_election.cc:302] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f46ac12998d3434abe3581d2dd12f080; no voters:
I20260620 19:09:40.633507 23510 leader_election.cc:288] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:40.633773 23520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:40.634174 23520 raft_consensus.cc:686] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [term 1 LEADER]: Becoming Leader. State: Replica: f46ac12998d3434abe3581d2dd12f080, State: Running, Role: LEADER
I20260620 19:09:40.634580 23520 consensus_queue.cc:237] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46ac12998d3434abe3581d2dd12f080" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38095 } }
I20260620 19:09:40.635385 23510 sys_catalog.cc:564] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:40.636552 23521 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f46ac12998d3434abe3581d2dd12f080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46ac12998d3434abe3581d2dd12f080" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38095 } } }
I20260620 19:09:40.636700 23521 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:40.636965 23522 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f46ac12998d3434abe3581d2dd12f080. Latest consensus state: current_term: 1 leader_uuid: "f46ac12998d3434abe3581d2dd12f080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46ac12998d3434abe3581d2dd12f080" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 38095 } } }
I20260620 19:09:40.637065 23522 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:40.637198 23529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:40.640457 23529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:40.648286 23529 catalog_manager.cc:1347] Generated new cluster ID: 13bc60fcce204e52baa036db4c864af3
I20260620 19:09:40.648399 23529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:40.657300 23529 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:40.657677 23529 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:40.664409 23529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f46ac12998d3434abe3581d2dd12f080: Generated new TSK 0
I20260620 19:09:40.665446 23529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:40.678987 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 23442
2026-06-20T19:09:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1484 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:40 dist-test-slave-6wrf krb5kdc[23541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:40 dist-test-slave-6wrf krb5kdc[23541](info): set up 2 sockets
Jun 20 19:09:40 dist-test-slave-6wrf krb5kdc[23541](info): commencing operation
krb5kdc: starting...
W20260620 19:09:41.794445 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:41 dist-test-slave-6wrf krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:09:41.955559 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:36727
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:46841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:36727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:42.100673 23554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:42.101022 23554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:42.101076 23554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:42.106305 23554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:42.106416 23554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:42.106441 23554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:42.106463 23554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:42.106482 23554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:42.113001 23554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:46841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:36727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:36727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:42.114357 23554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:42.115437 23554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:42.125494 23562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:42.125425 23559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:42.125440 23560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:42.125896 23554 server_base.cc:1034] running on GCE node
I20260620 19:09:42.126570 23554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:42.127740 23554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:42.129032 23554 hybrid_clock.cc:648] HybridClock initialized: now 1781982582128987 us; error 30 us; skew 500 ppm
Jun 20 19:09:42 dist-test-slave-6wrf krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982582, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:42.136161 23554 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:09:42.137362 23554 webserver.cc:460] Webserver started at http://127.1.154.190:36499/ using document root <none> and password file <none>
I20260620 19:09:42.138032 23554 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:42.138131 23554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:42.138425 23554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:42.140636 23554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "212f1dd46fd94a8b9ff9cd5def0b40f4"
format_stamp: "Formatted at 2026-06-20 19:09:42 on dist-test-slave-6wrf"
server_key: "1e1dcf4b002331a79593c965cbe739e9"
server_key_iv: "31ba0cf1a3c6e2f54b50a853a3a375c9"
server_key_version: "encryptionkey@0"
I20260620 19:09:42.142097 23554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "212f1dd46fd94a8b9ff9cd5def0b40f4"
format_stamp: "Formatted at 2026-06-20 19:09:42 on dist-test-slave-6wrf"
server_key: "1e1dcf4b002331a79593c965cbe739e9"
server_key_iv: "31ba0cf1a3c6e2f54b50a853a3a375c9"
server_key_version: "encryptionkey@0"
I20260620 19:09:42.147966 23554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 19:09:42.150818 23569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:42.152581 23554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:42.152755 23554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "212f1dd46fd94a8b9ff9cd5def0b40f4"
format_stamp: "Formatted at 2026-06-20 19:09:42 on dist-test-slave-6wrf"
server_key: "1e1dcf4b002331a79593c965cbe739e9"
server_key_iv: "31ba0cf1a3c6e2f54b50a853a3a375c9"
server_key_version: "encryptionkey@0"
I20260620 19:09:42.152894 23554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:42.181828 23554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:42.186497 23554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:42.186741 23554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:42.197654 23554 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:36727
I20260620 19:09:42.197708 23621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:36727 every 8 connection(s)
I20260620 19:09:42.198773 23554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:42.201674 23622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:42.205888 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 23554
I20260620 19:09:42.206036 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:42.206357 1642 external_mini_cluster.cc:1428] Setting key 3437e5612a091b8dbfb9e34fe1cd13c3
I20260620 19:09:42.207872 23622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4: Bootstrap starting.
I20260620 19:09:42.210156 23622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:42.211102 23622 log.cc:824] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:42.213310 23622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4: No bootstrap required, opened a new log
I20260620 19:09:42.217226 23622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212f1dd46fd94a8b9ff9cd5def0b40f4" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36727 } }
I20260620 19:09:42.217475 23622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Jun 20 19:09:42 dist-test-slave-6wrf krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:42.217564 23622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212f1dd46fd94a8b9ff9cd5def0b40f4, State: Initialized, Role: FOLLOWER
I20260620 19:09:42.218119 23622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212f1dd46fd94a8b9ff9cd5def0b40f4" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36727 } }
I20260620 19:09:42.218297 23622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:42.218341 23622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:42.218463 23622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:42.219664 23622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212f1dd46fd94a8b9ff9cd5def0b40f4" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36727 } }
I20260620 19:09:42.220084 23622 leader_election.cc:302] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 212f1dd46fd94a8b9ff9cd5def0b40f4; no voters:
I20260620 19:09:42.220458 23622 leader_election.cc:288] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:42.220657 23627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:42.221002 23627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [term 1 LEADER]: Becoming Leader. State: Replica: 212f1dd46fd94a8b9ff9cd5def0b40f4, State: Running, Role: LEADER
I20260620 19:09:42.221899 23622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:42.222100 23627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212f1dd46fd94a8b9ff9cd5def0b40f4" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36727 } }
I20260620 19:09:42.224630 23629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 212f1dd46fd94a8b9ff9cd5def0b40f4. Latest consensus state: current_term: 1 leader_uuid: "212f1dd46fd94a8b9ff9cd5def0b40f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212f1dd46fd94a8b9ff9cd5def0b40f4" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36727 } } }
I20260620 19:09:42.224867 23628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "212f1dd46fd94a8b9ff9cd5def0b40f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212f1dd46fd94a8b9ff9cd5def0b40f4" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 36727 } } }
I20260620 19:09:42.226601 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:42.208156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41975 (local address 127.1.154.190:36727)
0620 19:09:42.208563 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:42.208585 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:42.208626 (+ 41us) server_negotiation.cc:407] Connection header received
0620 19:09:42.209575 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:42.209780 (+ 205us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:42.210163 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:42.210505 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:42.211454 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:42.212826 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:42.214208 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:42.214887 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:42.215063 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:42.218436 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:42.218448 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:42.218479 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:42.218551 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:42.222251 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:42.222881 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:42.222890 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:42.222899 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:42.223064 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:42.223734 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:42.223737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:42.223739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:42.224303 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:42.224496 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:42.225279 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 19:09:42.225679 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":129,"threads_started":1}
I20260620 19:09:42.226900 23629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:42.226900 23628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4 [sys.catalog]: This master's current role is: LEADER
W20260620 19:09:42.228518 23641 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:09:42.228597 23641 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:09:42.228869 23644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:42.229835 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:42.229974 23644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:42.237479 23644 catalog_manager.cc:1347] Generated new cluster ID: c3278e61052c4105860e0adabaf0549e
I20260620 19:09:42.237648 23644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:42.243011 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:42.231840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40203 (local address 127.1.154.190:36727)
0620 19:09:42.231951 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:42.231955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:42.234620 (+ 2665us) server_negotiation.cc:407] Connection header received
0620 19:09:42.234807 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:42.234813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:42.234876 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:42.234968 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:42.235746 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:42.236288 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:42.237076 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:42.237327 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:42.237443 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:42.239583 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:42.239586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:42.239589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:42.239613 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:42.241277 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:42.241799 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:42.241802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:42.241804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:42.241882 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:42.242177 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:42.242180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:42.242182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:42.242367 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:42.242476 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:42.242758 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 19:09:42.242863 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:09:42.253475 23644 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:42.255323 23644 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:42.265971 23644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 212f1dd46fd94a8b9ff9cd5def0b40f4: Generated new TSK 0
I20260620 19:09:42.266747 23644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:42.276669 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 23554
2026-06-20T19:09:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1600 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:42 dist-test-slave-6wrf krb5kdc[23653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:42 dist-test-slave-6wrf krb5kdc[23653](info): set up 2 sockets
Jun 20 19:09:42 dist-test-slave-6wrf krb5kdc[23653](info): commencing operation
krb5kdc: starting...
W20260620 19:09:43.362063 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:43 dist-test-slave-6wrf krb5kdc[23653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:09:43.490463 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:37821
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:37821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:43.606688 23666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:43.606982 23666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:43.607025 23666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:43.611511 23666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:43.611654 23666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:43.611678 23666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:43.611699 23666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:43.611717 23666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:43.618731 23666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:37821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:37821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:43.619935 23666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:43.621436 23666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:43.632442 23671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:43.632438 23674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:43.632423 23672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:43.633000 23666 server_base.cc:1034] running on GCE node
I20260620 19:09:43.633725 23666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:43.635010 23666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:43.636248 23666 hybrid_clock.cc:648] HybridClock initialized: now 1781982583636225 us; error 48 us; skew 500 ppm
Jun 20 19:09:43 dist-test-slave-6wrf krb5kdc[23653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:43.640121 23666 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:09:43.641278 23666 webserver.cc:460] Webserver started at http://127.1.154.190:43329/ using document root <none> and password file <none>
I20260620 19:09:43.641976 23666 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:43.642082 23666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:43.642335 23666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:43.644692 23666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "4a98319e0689478d98b42f792f3601d8"
format_stamp: "Formatted at 2026-06-20 19:09:43 on dist-test-slave-6wrf"
server_key: "62e2120a6783c9e56ef7ee9edb0d1a96"
server_key_iv: "fd3b09c7321628fe99d09f4fa42e5ac4"
server_key_version: "encryptionkey@0"
I20260620 19:09:43.645320 23666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "4a98319e0689478d98b42f792f3601d8"
format_stamp: "Formatted at 2026-06-20 19:09:43 on dist-test-slave-6wrf"
server_key: "62e2120a6783c9e56ef7ee9edb0d1a96"
server_key_iv: "fd3b09c7321628fe99d09f4fa42e5ac4"
server_key_version: "encryptionkey@0"
I20260620 19:09:43.649421 23666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 19:09:43.651964 23681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:43.653378 23666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:09:43.653554 23666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "4a98319e0689478d98b42f792f3601d8"
format_stamp: "Formatted at 2026-06-20 19:09:43 on dist-test-slave-6wrf"
server_key: "62e2120a6783c9e56ef7ee9edb0d1a96"
server_key_iv: "fd3b09c7321628fe99d09f4fa42e5ac4"
server_key_version: "encryptionkey@0"
I20260620 19:09:43.653779 23666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:43.670284 23666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:43.674368 23666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:43.674691 23666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:43.684289 23666 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:37821
I20260620 19:09:43.684320 23733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:37821 every 8 connection(s)
I20260620 19:09:43.685277 23666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:43.688659 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 23666
I20260620 19:09:43.688839 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:43.689342 1642 external_mini_cluster.cc:1428] Setting key 48c838204da9e3cf44ddc4b4f12730bc
I20260620 19:09:43.689769 23734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:43.696156 23734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8: Bootstrap starting.
I20260620 19:09:43.698794 23734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:43.699894 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:43.690985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48329 (local address 127.1.154.190:37821)
0620 19:09:43.691463 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:43.691479 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:43.691516 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:09:43.692172 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:43.692331 (+ 159us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:43.692909 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:43.693442 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:43.694066 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:43.695345 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:43.696355 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:43.696830 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:43.697029 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:43.697978 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:43.698029 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:43.698042 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:43.698448 (+ 406us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:43.698507 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:43.698532 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:43.698675 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:43.698830 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 19:09:43.699157 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":168,"threads_started":1}
I20260620 19:09:43.700047 23734 log.cc:824] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:43.702769 23734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8: No bootstrap required, opened a new log
I20260620 19:09:43.702898 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:43.706741 23734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a98319e0689478d98b42f792f3601d8" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37821 } }
I20260620 19:09:43.706985 23734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:43.707048 23734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a98319e0689478d98b42f792f3601d8, State: Initialized, Role: FOLLOWER
I20260620 19:09:43.707521 23734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a98319e0689478d98b42f792f3601d8" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37821 } }
I20260620 19:09:43.707681 23734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:43.707768 23734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:43.707886 23734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:43.709216 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:43.704958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51665 (local address 127.1.154.190:37821)
0620 19:09:43.705085 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:43.705088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:43.705298 (+ 210us) server_negotiation.cc:407] Connection header received
0620 19:09:43.705444 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:43.705447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:43.705492 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:43.705596 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:43.705894 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:43.706430 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:43.707365 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:43.707761 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:43.707870 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:43.708739 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:43.708743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:43.708746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:43.708758 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:43.708786 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:43.708799 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:43.708874 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:43.708972 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 19:09:43.709039 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:09:43.709197 23734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a98319e0689478d98b42f792f3601d8" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37821 } }
I20260620 19:09:43.709532 23734 leader_election.cc:302] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a98319e0689478d98b42f792f3601d8; no voters:
I20260620 19:09:43.709887 23734 leader_election.cc:288] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:43.709959 23744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:43.710153 23744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [term 1 LEADER]: Becoming Leader. State: Replica: 4a98319e0689478d98b42f792f3601d8, State: Running, Role: LEADER
I20260620 19:09:43.710445 23744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a98319e0689478d98b42f792f3601d8" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37821 } }
I20260620 19:09:43.711180 23734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:43.711678 23745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a98319e0689478d98b42f792f3601d8. Latest consensus state: current_term: 1 leader_uuid: "4a98319e0689478d98b42f792f3601d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a98319e0689478d98b42f792f3601d8" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37821 } } }
I20260620 19:09:43.711859 23745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:43.712035 23746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a98319e0689478d98b42f792f3601d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a98319e0689478d98b42f792f3601d8" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 37821 } } }
I20260620 19:09:43.712159 23746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8 [sys.catalog]: This master's current role is: LEADER
W20260620 19:09:43.716115 23759 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:09:43.716203 23759 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:09:43.716290 23750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:43.717168 23750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:43.724146 23750 catalog_manager.cc:1347] Generated new cluster ID: 3600386316c74dd9bc8dcdc3174b8894
I20260620 19:09:43.724227 23750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:43.730046 23750 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:43.731144 23750 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:43.735643 23750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a98319e0689478d98b42f792f3601d8: Generated new TSK 0
I20260620 19:09:43.736392 23750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:43.740731 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 23666
2026-06-20T19:09:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:43 dist-test-slave-6wrf krb5kdc[23765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:43 dist-test-slave-6wrf krb5kdc[23765](info): set up 2 sockets
Jun 20 19:09:43 dist-test-slave-6wrf krb5kdc[23765](info): commencing operation
krb5kdc: starting...
W20260620 19:09:44.802627 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:44 dist-test-slave-6wrf krb5kdc[23765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:09:44.933158 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43921
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:43921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:45.046797 23778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:45.047083 23778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:45.047158 23778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:45.051257 23778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:45.051354 23778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:45.051394 23778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:45.051453 23778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:45.051493 23778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:45.056739 23778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:43921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:45.057843 23778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:45.058859 23778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:45.067939 23784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:45.067960 23783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:45.068070 23778 server_base.cc:1034] running on GCE node
W20260620 19:09:45.067960 23786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:45.068866 23778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:45.070065 23778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:45.071272 23778 hybrid_clock.cc:648] HybridClock initialized: now 1781982585071254 us; error 44 us; skew 500 ppm
Jun 20 19:09:45 dist-test-slave-6wrf krb5kdc[23765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982585, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:45.074683 23778 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:09:45.075563 23778 webserver.cc:460] Webserver started at http://127.1.154.190:38303/ using document root <none> and password file <none>
I20260620 19:09:45.076069 23778 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:45.076128 23778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:45.076368 23778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:45.078498 23778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "6f825e34dcd14da29edfe479f6ecff7d"
format_stamp: "Formatted at 2026-06-20 19:09:45 on dist-test-slave-6wrf"
server_key: "ac9aeddfdd37da18ed39306ed7fe410b"
server_key_iv: "376fb1ba527c9fd6682e5490e56a0e49"
server_key_version: "encryptionkey@0"
I20260620 19:09:45.078992 23778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "6f825e34dcd14da29edfe479f6ecff7d"
format_stamp: "Formatted at 2026-06-20 19:09:45 on dist-test-slave-6wrf"
server_key: "ac9aeddfdd37da18ed39306ed7fe410b"
server_key_iv: "376fb1ba527c9fd6682e5490e56a0e49"
server_key_version: "encryptionkey@0"
I20260620 19:09:45.083302 23778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:09:45.086709 23793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:45.088356 23778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:09:45.088522 23778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "6f825e34dcd14da29edfe479f6ecff7d"
format_stamp: "Formatted at 2026-06-20 19:09:45 on dist-test-slave-6wrf"
server_key: "ac9aeddfdd37da18ed39306ed7fe410b"
server_key_iv: "376fb1ba527c9fd6682e5490e56a0e49"
server_key_version: "encryptionkey@0"
I20260620 19:09:45.088649 23778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:45.093237 23778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:45.096490 23778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:45.096684 23778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:45.103936 23778 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:43921
I20260620 19:09:45.103961 23845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:43921 every 8 connection(s)
I20260620 19:09:45.104857 23778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:45.107573 23846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:45.110231 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 23778
I20260620 19:09:45.110384 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:45.110690 1642 external_mini_cluster.cc:1428] Setting key 86b0c7f5f71df032c7131a44fdd46b21
I20260620 19:09:45.113847 23846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d: Bootstrap starting.
I20260620 19:09:45.116638 23849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:45.112106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40061 (local address 127.1.154.190:43921)
0620 19:09:45.112664 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:45.112689 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:45.112739 (+ 50us) server_negotiation.cc:407] Connection header received
0620 19:09:45.113697 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:45.113719 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:45.114089 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:45.114488 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:45.115240 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:45.115246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:45.115262 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:45.115277 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:45.115351 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:45.115379 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:45.115515 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:45.115585 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 19:09:45.115732 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":129,"threads_started":1}
I20260620 19:09:45.116701 23846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:45.117631 23846 log.cc:824] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:45.119678 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:45.120005 23846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d: No bootstrap required, opened a new log
I20260620 19:09:45.123440 23849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:45.121831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42681 (local address 127.1.154.190:43921)
0620 19:09:45.121944 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:45.121948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:45.122192 (+ 244us) server_negotiation.cc:407] Connection header received
0620 19:09:45.122285 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:45.122287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:45.122317 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:45.122394 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:45.123102 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:45.123105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:45.123107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:09:45.123117 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:45.123141 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:09:45.123150 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:45.123226 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:45.123323 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 19:09:45.123352 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:09:45.124908 23846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f825e34dcd14da29edfe479f6ecff7d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43921 } }
I20260620 19:09:45.125159 23846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:45.125253 23846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f825e34dcd14da29edfe479f6ecff7d, State: Initialized, Role: FOLLOWER
I20260620 19:09:45.125732 23846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f825e34dcd14da29edfe479f6ecff7d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43921 } }
I20260620 19:09:45.125891 23846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:45.125934 23846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:45.126027 23846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:45.127080 23846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f825e34dcd14da29edfe479f6ecff7d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43921 } }
I20260620 19:09:45.127419 23846 leader_election.cc:302] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f825e34dcd14da29edfe479f6ecff7d; no voters:
I20260620 19:09:45.127737 23846 leader_election.cc:288] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:45.128038 23856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:45.128441 23856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [term 1 LEADER]: Becoming Leader. State: Replica: 6f825e34dcd14da29edfe479f6ecff7d, State: Running, Role: LEADER
I20260620 19:09:45.128855 23856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f825e34dcd14da29edfe479f6ecff7d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43921 } }
I20260620 19:09:45.129093 23846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:45.131067 23858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f825e34dcd14da29edfe479f6ecff7d. Latest consensus state: current_term: 1 leader_uuid: "6f825e34dcd14da29edfe479f6ecff7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f825e34dcd14da29edfe479f6ecff7d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43921 } } }
I20260620 19:09:45.131565 23858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:45.131300 23857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f825e34dcd14da29edfe479f6ecff7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f825e34dcd14da29edfe479f6ecff7d" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43921 } } }
I20260620 19:09:45.131860 23857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d [sys.catalog]: This master's current role is: LEADER
W20260620 19:09:45.133800 23870 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:09:45.133862 23870 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:09:45.133936 23871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:45.134613 23871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:45.140010 23871 catalog_manager.cc:1347] Generated new cluster ID: 25b5a6891bc6409ca53a08f4a63bdd1e
I20260620 19:09:45.140091 23871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:45.150312 23871 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:45.150650 23871 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:45.160147 23871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f825e34dcd14da29edfe479f6ecff7d: Generated new TSK 0
I20260620 19:09:45.160907 23871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:09:45.166080 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 23778
2026-06-20T19:09:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1425 ms)
[----------] 18 tests from AuthTokenIssuingTest (26543 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:45 dist-test-slave-6wrf krb5kdc[23877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:45 dist-test-slave-6wrf krb5kdc[23877](info): set up 2 sockets
Jun 20 19:09:45 dist-test-slave-6wrf krb5kdc[23877](info): commencing operation
krb5kdc: starting...
W20260620 19:09:46.226342 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:46 dist-test-slave-6wrf krb5kdc[23877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:09:46.357749 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:39143
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:39143,127.1.154.189:34069,127.1.154.188:42293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:46.468674 23890 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:09:46.469045 23890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:46.469082 23890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:46.469144 23890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:46.474881 23890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:46.474975 23890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:46.475023 23890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:46.475059 23890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:46.475090 23890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:46.481443 23890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:39143,127.1.154.189:34069,127.1.154.188:42293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:39143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:46.482607 23890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:46.483651 23890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:46.492178 23895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:46.492225 23898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:46.492417 23890 server_base.cc:1034] running on GCE node
W20260620 19:09:46.492178 23896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:46.493132 23890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:46.494256 23890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:46.495607 23890 hybrid_clock.cc:648] HybridClock initialized: now 1781982586495575 us; error 52 us; skew 500 ppm
Jun 20 19:09:46 dist-test-slave-6wrf krb5kdc[23877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:46.499092 23890 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:09:46.500113 23890 webserver.cc:460] Webserver started at http://127.1.154.190:44057/ using document root <none> and password file <none>
I20260620 19:09:46.500655 23890 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:46.500713 23890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:46.500880 23890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:46.502990 23890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "480d2acde73040ec9deaedad1fcb91ab"
format_stamp: "Formatted at 2026-06-20 19:09:46 on dist-test-slave-6wrf"
server_key: "6adf0bbc45a848e64b7ee9039fd69739"
server_key_iv: "bfafbe8d9de7bbb360196bd2d5219e50"
server_key_version: "encryptionkey@0"
I20260620 19:09:46.503504 23890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "480d2acde73040ec9deaedad1fcb91ab"
format_stamp: "Formatted at 2026-06-20 19:09:46 on dist-test-slave-6wrf"
server_key: "6adf0bbc45a848e64b7ee9039fd69739"
server_key_iv: "bfafbe8d9de7bbb360196bd2d5219e50"
server_key_version: "encryptionkey@0"
I20260620 19:09:46.507452 23890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:09:46.509919 23905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:46.511195 23890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:09:46.511298 23890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "480d2acde73040ec9deaedad1fcb91ab"
format_stamp: "Formatted at 2026-06-20 19:09:46 on dist-test-slave-6wrf"
server_key: "6adf0bbc45a848e64b7ee9039fd69739"
server_key_iv: "bfafbe8d9de7bbb360196bd2d5219e50"
server_key_version: "encryptionkey@0"
I20260620 19:09:46.511410 23890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:46.523995 23890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:46.527154 23890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:46.527318 23890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:46.534157 23890 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:39143
I20260620 19:09:46.534628 23957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:39143 every 8 connection(s)
I20260620 19:09:46.535117 23890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:46.538118 23958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:46.540946 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 23890
I20260620 19:09:46.541047 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:46.541323 1642 external_mini_cluster.cc:1428] Setting key 40f521966f8262cc6154c329b5fcbd13
I20260620 19:09:46.542184 23958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.154.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.189@KRBTEST.COM" created.
Jun 20 19:09:46 dist-test-slave-6wrf krb5kdc[23877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:46.560487 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.544997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39143 (local address 127.0.0.1:46780)
0620 19:09:46.545639 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.545689 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:46.546583 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:46.547283 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:46.547288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:46.547752 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:46.548030 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:46.548044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.549321 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.549324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:46.550319 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:46.550331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.551071 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.551075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:46.551198 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:46.551953 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:46.551981 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:46.553755 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:46.556324 (+ 2569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:46.556331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:46.556340 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:46.556609 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:46.557118 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:46.557120 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:46.557122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:46.557244 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:46.558927 (+ 1683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:46.558934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:46.559120 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 19:09:46.559291 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 19:09:46.559601 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":32,"spinlock_wait_cycles":25984,"thread_start_us":155,"threads_started":1}
I20260620 19:09:46.560491 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.544997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.1.154.190:39143)
0620 19:09:46.545458 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:46.545478 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:46.545800 (+ 322us) server_negotiation.cc:407] Connection header received
0620 19:09:46.546905 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:46.546910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:46.547092 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:46.547287 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:46.548191 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.549169 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.550448 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.550904 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.551126 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:46.553929 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:46.553938 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:46.553957 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:46.553990 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:46.556201 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:46.556752 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:46.556763 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:46.556768 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:46.556923 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:46.557587 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:46.557626 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:46.557631 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:46.558680 (+ 1049us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:46.559018 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:46.559422 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 19:09:46.559715 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":296,"thread_start_us":159,"threads_started":1}
I20260620 19:09:46.563815 23958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:09:46.565120 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.564728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:34069 (local address 127.0.0.1:51858)
0620 19:09:46.564891 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.564989 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:34069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 19:09:46.565495 23908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.154.189:34069: connect: Connection refused (error 111)
I20260620 19:09:46.566807 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.566504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:34069 (local address 127.0.0.1:51874)
0620 19:09:46.566677 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.566756 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:34069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260620 19:09:46.567020 23958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.189:34069: Network error: Client connection negotiation failed: client connection to 127.1.154.189:34069: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.189.keytab.
Entry for principal kudu/127.1.154.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.189.keytab.
I20260620 19:09:46.573545 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.189:34069
--webserver_interface=127.1.154.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.189.keytab
--principal=kudu/127.1.154.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:39143,127.1.154.189:34069,127.1.154.188:42293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:09:46.616324 23958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } attempt: 1
I20260620 19:09:46.617547 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.616968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:34069 (local address 127.0.0.1:51876)
0620 19:09:46.617283 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.617389 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:34069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260620 19:09:46.618665 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.618317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:34069 (local address 127.0.0.1:51884)
0620 19:09:46.618499 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.618590 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:34069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 19:09:46.618984 23958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.189:34069: Network error: Client connection negotiation failed: client connection to 127.1.154.189:34069: connect: Connection refused (error 111)
W20260620 19:09:46.684969 23965 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:09:46.685277 23965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:46.685300 23965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:46.685392 23965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 19:09:46.687300 23958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } attempt: 2
I20260620 19:09:46.688371 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.687899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:34069 (local address 127.0.0.1:51888)
0620 19:09:46.688173 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.688255 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:34069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 19:09:46.689225 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.688989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:34069 (local address 127.0.0.1:51896)
0620 19:09:46.689127 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.689174 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:34069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 19:09:46.689378 23958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.189:34069: Network error: Client connection negotiation failed: client connection to 127.1.154.189:34069: connect: Connection refused (error 111)
W20260620 19:09:46.690308 23965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:46.690369 23965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:46.690389 23965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:46.690407 23965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:46.690423 23965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:46.695823 23965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:39143,127.1.154.189:34069,127.1.154.188:42293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.189.keytab
--principal=kudu/127.1.154.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.189:34069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.154.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:46.696911 23965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:46.698002 23965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:46.706900 23970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:46.706902 23971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:46.706902 23973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:46.707327 23965 server_base.cc:1034] running on GCE node
I20260620 19:09:46.707844 23965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:46.708891 23965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:46.710095 23965 hybrid_clock.cc:648] HybridClock initialized: now 1781982586710078 us; error 35 us; skew 500 ppm
Jun 20 19:09:46 dist-test-slave-6wrf krb5kdc[23877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:46.712908 23965 init.cc:377] Logged in from keytab as kudu/127.1.154.189@KRBTEST.COM (short username kudu)
I20260620 19:09:46.714035 23965 webserver.cc:460] Webserver started at http://127.1.154.189:43037/ using document root <none> and password file <none>
I20260620 19:09:46.714704 23965 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:46.714802 23965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:46.715060 23965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:46.717317 23965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/data/instance:
uuid: "0ea65089539a4cf39b5e969c013db61f"
format_stamp: "Formatted at 2026-06-20 19:09:46 on dist-test-slave-6wrf"
server_key: "62c25e3e31b0ee944a6cbc686934878e"
server_key_iv: "531ded51ec58e91bccad016d26f944f4"
server_key_version: "encryptionkey@0"
I20260620 19:09:46.717943 23965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/wal/instance:
uuid: "0ea65089539a4cf39b5e969c013db61f"
format_stamp: "Formatted at 2026-06-20 19:09:46 on dist-test-slave-6wrf"
server_key: "62c25e3e31b0ee944a6cbc686934878e"
server_key_iv: "531ded51ec58e91bccad016d26f944f4"
server_key_version: "encryptionkey@0"
I20260620 19:09:46.722383 23965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:09:46.724944 23980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:46.726581 23965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:09:46.726745 23965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/wal
uuid: "0ea65089539a4cf39b5e969c013db61f"
format_stamp: "Formatted at 2026-06-20 19:09:46 on dist-test-slave-6wrf"
server_key: "62c25e3e31b0ee944a6cbc686934878e"
server_key_iv: "531ded51ec58e91bccad016d26f944f4"
server_key_version: "encryptionkey@0"
I20260620 19:09:46.726866 23965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:46.749763 23965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:46.753360 23965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:46.753729 23965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:46.761785 23965 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.189:34069
I20260620 19:09:46.762061 24032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.189:34069 every 8 connection(s)
I20260620 19:09:46.762804 23965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/data/info.pb
I20260620 19:09:46.765547 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:46.769305 24033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:09:46.770857 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 23965
I20260620 19:09:46.771015 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-1/wal/instance
I20260620 19:09:46.771350 1642 external_mini_cluster.cc:1428] Setting key 48e874141b9ac4be60469642431eada4
Jun 20 19:09:46 dist-test-slave-6wrf krb5kdc[23877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.189@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.154.188@KRBTEST.COM; defaulting to no policy
I20260620 19:09:46.780597 23958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.188@KRBTEST.COM" created.
I20260620 19:09:46.785545 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.771460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.1.154.190:39143)
0620 19:09:46.771611 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:46.771614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:46.772673 (+ 1059us) server_negotiation.cc:407] Connection header received
0620 19:09:46.773659 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:46.773676 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:46.773765 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:46.773857 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:46.775004 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.775537 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.776689 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.776939 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.777057 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:46.780910 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:46.780916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:46.780920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:46.780967 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:46.783006 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:46.783629 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:46.783633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:46.783635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:46.783721 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:46.784168 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:46.784171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:46.784173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:46.784431 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:46.784546 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:46.785230 (+ 684us) server_negotiation.cc:299] Negotiation successful
0620 19:09:46.785355 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:09:46.787288 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.771925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39143 (local address 127.0.0.1:46796)
0620 19:09:46.772497 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.772551 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:46.773434 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:46.774040 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:46.774047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:46.774388 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:46.774865 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:46.774878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.775683 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.775685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:46.776565 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:46.776574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.777099 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.777103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:46.777276 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:46.778334 (+ 1058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:46.778352 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:46.780504 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:46.783160 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:46.783167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:46.783189 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:46.783481 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:46.783831 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:46.783834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:46.783836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:46.784048 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:46.784562 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:46.784568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:46.784937 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 19:09:46.785812 (+ 875us) client_negotiation.cc:240] Negotiation successful
0620 19:09:46.786162 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":179,"threads_started":1}
Jun 20 19:09:46 dist-test-slave-6wrf krb5kdc[23877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for kudu/127.1.154.189@KRBTEST.COM
I20260620 19:09:46.790827 24033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:09:46.792650 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.781206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:34069 (local address 127.0.0.1:51902)
0620 19:09:46.781820 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.781836 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:46.781929 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:46.782478 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:46.782485 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:46.782670 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:46.782817 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:46.782826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.784047 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.784053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:46.784801 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:46.784812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.785923 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.785927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:46.786031 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:46.786558 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:46.786573 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:46.788202 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:46.790735 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:46.790739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:46.790741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:46.790979 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:46.791339 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:46.791343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:46.791345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:46.791451 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:46.792062 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:46.792065 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:46.792193 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 19:09:46.792295 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 19:09:46.792471 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496}
I20260620 19:09:46.793366 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.781443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51902 (local address 127.1.154.189:34069)
0620 19:09:46.781976 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:46.781985 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:46.782034 (+ 49us) server_negotiation.cc:407] Connection header received
0620 19:09:46.782090 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:46.782097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:46.782324 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:46.782452 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:46.783012 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.783864 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.784949 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.785498 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.786240 (+ 742us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:46.788342 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:46.788356 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:46.788365 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:46.788386 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:46.790573 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:46.791110 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:46.791115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:46.791116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:46.791223 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:46.791581 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:46.791585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:46.791586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:46.791922 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:46.792996 (+ 1074us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:46.793104 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 19:09:46.793228 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":188,"threads_started":1}
I20260620 19:09:46.795889 23958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:09:46.796761 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.796460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45348)
0620 19:09:46.796612 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.796678 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 19:09:46.797778 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.797253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45360)
0620 19:09:46.797570 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.797682 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
W20260620 19:09:46.797976 23958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:42293: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Jun 20 19:09:46 dist-test-slave-6wrf krb5kdc[23877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.189@KRBTEST.COM for kudu/127.1.154.189@KRBTEST.COM
I20260620 19:09:46.803097 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.791400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:34069 (local address 127.0.0.1:51908)
0620 19:09:46.791519 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.791532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:46.791605 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:46.792036 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:46.792039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:46.792201 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:46.792278 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:46.792283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.793458 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.793465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:46.794544 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:46.794557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.796294 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.796303 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:46.796400 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:46.797019 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:46.797031 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:46.798598 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:46.800958 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:46.800963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:46.800965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:46.801263 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:46.801736 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:46.801738 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:46.801740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:46.801830 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:46.802386 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:46.802390 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:46.802558 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 19:09:46.802703 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 19:09:46.802930 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 19:09:46.803364 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.791532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51908 (local address 127.1.154.189:34069)
0620 19:09:46.791834 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:46.791840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:46.791849 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:09:46.791881 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:46.791883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:46.791913 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:46.792018 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:46.792387 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.793274 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.794883 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:46.796142 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:46.796251 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:46.798822 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:46.798825 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:46.798827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:46.798849 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:46.800779 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:46.801471 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:46.801476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:46.801477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:46.801572 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:46.801977 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:46.801980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:46.801981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:46.802190 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:46.802282 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:46.802841 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 19:09:46.803061 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":118,"threads_started":1}
I20260620 19:09:46.804438 24033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.188.keytab.
Entry for principal kudu/127.1.154.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.188.keytab.
I20260620 19:09:46.805709 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.804932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45368)
0620 19:09:46.805337 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.805502 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295}
W20260620 19:09:46.806015 23983 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
I20260620 19:09:46.806656 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.188:42293
--webserver_interface=127.1.154.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.188.keytab
--principal=kudu/127.1.154.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:39143,127.1.154.189:34069,127.1.154.188:42293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:09:46.807341 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.806899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45374)
0620 19:09:46.807153 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.807237 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260620 19:09:46.807548 24033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:42293: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
I20260620 19:09:46.829270 23958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } attempt: 1
I20260620 19:09:46.830235 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.829898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45382)
0620 19:09:46.830066 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.830136 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 19:09:46.831058 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.830860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45398)
0620 19:09:46.830948 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.831001 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 19:09:46.831251 23958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:42293: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
I20260620 19:09:46.856927 24033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } attempt: 1
I20260620 19:09:46.858101 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.857793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45400)
0620 19:09:46.857949 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.858018 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 19:09:46.858829 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.858652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45412)
0620 19:09:46.858741 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.858783 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 19:09:46.858974 24033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:42293: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
I20260620 19:09:46.906442 23958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } attempt: 2
I20260620 19:09:46.907627 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.907212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45428)
0620 19:09:46.907431 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.907524 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 19:09:46.908576 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.908334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45440)
0620 19:09:46.908456 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.908509 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 19:09:46.908761 23958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:42293: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
W20260620 19:09:46.926528 24041 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:09:46.926873 24041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:46.926903 24041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:46.926950 24041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 19:09:46.927206 24033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } attempt: 2
I20260620 19:09:46.928448 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.927760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45454)
0620 19:09:46.928215 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.928327 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260620 19:09:46.929463 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:46.929155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45468)
0620 19:09:46.929316 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:46.929387 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 19:09:46.929672 24033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:42293: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
W20260620 19:09:46.932428 24041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:46.932499 24041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:46.932523 24041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:46.932543 24041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:46.932562 24041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:46.939471 24041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:39143,127.1.154.189:34069,127.1.154.188:42293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.188.keytab
--principal=kudu/127.1.154.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.188:42293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.154.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:46.941067 24041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:46.942641 24041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:46.953866 24047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:46.953893 24049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:46.953866 24046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:46.954155 24041 server_base.cc:1034] running on GCE node
I20260620 19:09:46.954862 24041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:46.956032 24041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:46.959337 24041 hybrid_clock.cc:648] HybridClock initialized: now 1781982586959313 us; error 35 us; skew 500 ppm
Jun 20 19:09:46 dist-test-slave-6wrf krb5kdc[23877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:46.962608 24041 init.cc:377] Logged in from keytab as kudu/127.1.154.188@KRBTEST.COM (short username kudu)
I20260620 19:09:46.963577 24041 webserver.cc:460] Webserver started at http://127.1.154.188:44131/ using document root <none> and password file <none>
I20260620 19:09:46.964175 24041 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:46.964272 24041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:46.964507 24041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:46.966694 24041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/data/instance:
uuid: "f10f11b001b64121920449e87e5c8463"
format_stamp: "Formatted at 2026-06-20 19:09:46 on dist-test-slave-6wrf"
server_key: "70c8528f854c6c361bf71ebe8e2c7e00"
server_key_iv: "8d7c336e9ab83f5c1119dc663ade7e4c"
server_key_version: "encryptionkey@0"
I20260620 19:09:46.967789 24041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/wal/instance:
uuid: "f10f11b001b64121920449e87e5c8463"
format_stamp: "Formatted at 2026-06-20 19:09:46 on dist-test-slave-6wrf"
server_key: "70c8528f854c6c361bf71ebe8e2c7e00"
server_key_iv: "8d7c336e9ab83f5c1119dc663ade7e4c"
server_key_version: "encryptionkey@0"
I20260620 19:09:46.973531 24041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 19:09:46.976836 24056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:46.978936 24041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:46.979192 24041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/wal
uuid: "f10f11b001b64121920449e87e5c8463"
format_stamp: "Formatted at 2026-06-20 19:09:46 on dist-test-slave-6wrf"
server_key: "70c8528f854c6c361bf71ebe8e2c7e00"
server_key_iv: "8d7c336e9ab83f5c1119dc663ade7e4c"
server_key_version: "encryptionkey@0"
I20260620 19:09:46.979375 24041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:47.008096 23958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } attempt: 3
I20260620 19:09:47.009419 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.008909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45480)
0620 19:09:47.009207 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:47.009300 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260620 19:09:47.010365 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.010147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45486)
0620 19:09:47.010259 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:47.010316 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 19:09:47.010502 23958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:42293: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
I20260620 19:09:47.012441 24041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:47.015771 24041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:47.016007 24041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:47.020880 24033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } attempt: 3
I20260620 19:09:47.021788 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.021413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45502)
0620 19:09:47.021638 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:47.021708 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 19:09:47.022576 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.022340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45504)
0620 19:09:47.022459 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:47.022506 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 19:09:47.022718 24033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:42293: Network error: Client connection negotiation failed: client connection to 127.1.154.188:42293: connect: Connection refused (error 111)
I20260620 19:09:47.024025 24041 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.188:42293
I20260620 19:09:47.024302 24108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.188:42293 every 8 connection(s)
I20260620 19:09:47.025157 24041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/data/info.pb
I20260620 19:09:47.028023 24109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:47.031981 24109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:09:47.035387 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 24041
I20260620 19:09:47.035538 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982293404112-1642-0/minicluster-data/master-2/wal/instance
I20260620 19:09:47.035794 1642 external_mini_cluster.cc:1428] Setting key 5ae278a5af66461c31dd3494a406542a
Jun 20 19:09:47 dist-test-slave-6wrf krb5kdc[23877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.188@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
Jun 20 19:09:47 dist-test-slave-6wrf krb5kdc[23877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.188@KRBTEST.COM
I20260620 19:09:47.047492 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.033753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.1.154.190:39143)
0620 19:09:47.033905 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.033909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.034750 (+ 841us) server_negotiation.cc:407] Connection header received
0620 19:09:47.035741 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.035745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.035807 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.035920 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.037416 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.037979 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.039135 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.039410 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.039490 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.042253 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:47.042260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:47.042266 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:47.042300 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:47.044271 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.045391 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.045398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.045400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.045701 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.046177 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.046182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.046186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.046479 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:47.046587 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:47.047154 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 19:09:47.047299 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:09:47.048429 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.034084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:39143 (local address 127.0.0.1:46802)
0620 19:09:47.034552 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:47.034614 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:47.035466 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:47.036081 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:47.036094 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:47.036542 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:47.037222 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:47.037265 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.038114 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.038118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:47.039001 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:47.039011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.039565 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.039569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:47.039785 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.040430 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:47.040445 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:47.042100 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:47.044623 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:47.044646 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:47.044665 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:47.045206 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:47.045823 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:47.045828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:47.045831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:47.046056 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:47.046598 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:47.046604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:47.046877 (+ 273us) client_negotiation.cc:769] Sending connection context
0620 19:09:47.047093 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 19:09:47.047419 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"spinlock_wait_cycles":2816,"thread_start_us":118,"threads_started":1}
I20260620 19:09:47.049091 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.036471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.1.154.188:42293)
0620 19:09:47.036820 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.036867 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.036887 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:09:47.037018 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.037022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.037156 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.037276 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.038281 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.039146 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.040311 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.040672 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.040752 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.044077 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:47.044084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:47.044099 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:47.044129 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:47.046886 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.047307 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.047314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.047321 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.047394 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.047720 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.047723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.047725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.047987 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:47.048143 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:47.048451 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 19:09:47.048911 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":266,"spinlock_wait_cycles":1792,"thread_start_us":138,"threads_started":1}
I20260620 19:09:47.049916 24109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:09:47 dist-test-slave-6wrf krb5kdc[23877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.188@KRBTEST.COM for kudu/127.1.154.189@KRBTEST.COM
Jun 20 19:09:47 dist-test-slave-6wrf krb5kdc[23877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:47.060735 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.050401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:34069 (local address 127.0.0.1:51920)
0620 19:09:47.050654 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:47.050687 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:47.050774 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:47.051119 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:47.051122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:47.051323 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:47.051446 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:47.051455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.052902 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.052905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:47.053769 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:47.053779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.054527 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.054532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:47.054678 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.055360 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:47.055379 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:47.056667 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:47.058731 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:47.058739 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:47.058744 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:47.059099 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:47.059564 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:47.059570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:47.059575 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:47.059680 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:47.060213 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:47.060215 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:47.060299 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 19:09:47.060399 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 19:09:47.060523 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260620 19:09:47.060801 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.050538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.1.154.189:34069)
0620 19:09:47.050682 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.050688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.050765 (+ 77us) server_negotiation.cc:407] Connection header received
0620 19:09:47.050882 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.050885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.050934 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.051027 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.051694 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.052752 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.053944 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.054310 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.054427 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.056814 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:47.056820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:47.056827 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:47.056859 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:47.058575 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.059272 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.059275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.059276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.059399 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.059812 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.059817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.059819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.060095 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:47.060221 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:47.060487 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 19:09:47.060650 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:09:47.061590 24109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:09:47.062470 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.051724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.1.154.190:39143)
0620 19:09:47.051888 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.051892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.051902 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:09:47.051931 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.051933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.051966 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.052031 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.053038 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.053822 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.054846 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.055388 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.055498 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.058219 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:47.058223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:47.058226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:47.058263 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:47.060477 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.061037 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.061040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.061048 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.061149 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.061498 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.061503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.061504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.061760 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:47.062011 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:47.062178 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 19:09:47.062313 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 19:09:47 dist-test-slave-6wrf krb5kdc[23877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.188@KRBTEST.COM for kudu/127.1.154.188@KRBTEST.COM
Jun 20 19:09:47 dist-test-slave-6wrf krb5kdc[23877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.189@KRBTEST.COM
I20260620 19:09:47.070324 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.062201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45532)
0620 19:09:47.062405 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:47.062419 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:47.062490 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:47.062785 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:47.062788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:47.062950 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:47.063017 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:47.063045 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.064128 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.064131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:47.064620 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:47.064627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.065096 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.065099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:47.065295 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.065807 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:47.065819 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:47.067044 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:47.068725 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:47.068731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:47.068733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:47.068946 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:47.069315 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:47.069317 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:47.069318 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:47.069388 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:47.069809 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:47.069813 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:47.069896 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 19:09:47.070038 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 19:09:47.070178 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":66}
I20260620 19:09:47.070513 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.062306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.1.154.188:42293)
0620 19:09:47.062428 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.062435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.062455 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:09:47.062591 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.062595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.062634 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.062696 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.063171 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.063895 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.064736 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.064991 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.065100 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.067181 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:47.067184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:47.067186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:47.067206 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:47.068620 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.069043 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.069047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.069051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.069130 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.069488 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.069492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.069495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.069713 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:47.069828 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:47.070188 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 19:09:47.070310 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":45}
I20260620 19:09:47.072768 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.063670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.1.154.189:34069)
0620 19:09:47.063790 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.063793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.063827 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:09:47.063957 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.063959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.063990 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.064048 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.064625 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.065112 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.066059 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.066303 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.066428 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.068815 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:47.068819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:47.068822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:47.068848 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:47.070941 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.071423 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.071426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.071428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.071513 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.071761 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.071764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.071766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.071979 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:47.072153 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:47.072282 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 19:09:47.072566 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:09:47.074515 24109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463: Bootstrap starting.
I20260620 19:09:47.077083 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:47.077353 24109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:47.078246 24109 log.cc:824] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:47.078457 23932 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "480d2acde73040ec9deaedad1fcb91ab"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46810
I20260620 19:09:47.080467 24109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463: No bootstrap required, opened a new log
I20260620 19:09:47.085390 24109 raft_consensus.cc:348] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } }
I20260620 19:09:47.085587 24109 raft_consensus.cc:729] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10f11b001b64121920449e87e5c8463, State: Initialized, Role: FOLLOWER
I20260620 19:09:47.085925 24109 consensus_queue.cc:260] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } }
I20260620 19:09:47.086427 24115 sys_catalog.cc:455] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } } }
I20260620 19:09:47.086567 24115 sys_catalog.cc:458] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:09:47.087203 24109 sys_catalog.cc:564] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 19:09:47.091857 24126 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:09:47.091943 24126 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:09:47.090955 24041 master_runner.cc:186] Error getting master registration for 127.1.154.190:39143: OK, instance_id { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" instance_seqno: 1781982586532190 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:09:47.093655 24041 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:09:47.165846 24033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } attempt: 4
I20260620 19:09:47.174768 23958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } attempt: 4
Jun 20 19:09:47 dist-test-slave-6wrf krb5kdc[23877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.189@KRBTEST.COM for kudu/127.1.154.188@KRBTEST.COM
I20260620 19:09:47.179824 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.166491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45546)
0620 19:09:47.166703 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:47.166717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:47.166832 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:47.167090 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:47.167092 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:47.167295 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:47.167378 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:47.167383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.168136 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.168139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:47.168657 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:47.168664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.169114 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.169117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:47.169198 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.173978 (+ 4780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:47.174001 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:47.175470 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:47.178158 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:47.178161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:47.178164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:47.178407 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:47.178692 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:47.178695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:47.178697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:47.178779 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:47.179285 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:47.179288 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:47.179385 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 19:09:47.179508 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 19:09:47.179647 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260620 19:09:47.180361 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.166584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.1.154.188:42293)
0620 19:09:47.166707 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.166713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.166777 (+ 64us) server_negotiation.cc:407] Connection header received
0620 19:09:47.166926 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.166930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.166981 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.167072 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.167500 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.168019 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.168778 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.168998 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.169107 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.175637 (+ 6530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:47.175641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:47.175643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:47.175670 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:47.178027 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.178516 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.178519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.178521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.178599 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.178887 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.178890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.178892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.179149 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:47.179240 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:47.180104 (+ 864us) server_negotiation.cc:299] Negotiation successful
0620 19:09:47.180217 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 19:09:47 dist-test-slave-6wrf krb5kdc[23877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for kudu/127.1.154.188@KRBTEST.COM
I20260620 19:09:47.183964 24033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f: Bootstrap starting.
I20260620 19:09:47.186753 24033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:47.187595 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.175315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:42293 (local address 127.0.0.1:45558)
0620 19:09:47.175693 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:47.175709 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:47.175811 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:47.176133 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:47.176136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:47.176316 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:47.176425 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:47.176432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.178031 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.178035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:47.179064 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:47.179099 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.179869 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.179873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:47.179970 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.181166 (+ 1196us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:47.181189 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:47.183122 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:47.185859 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:47.185863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:47.185865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:47.186125 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:47.186461 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:47.186466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:47.186470 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:47.186561 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:47.186997 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:47.187000 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:47.187109 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 19:09:47.187272 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 19:09:47.187433 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289}
I20260620 19:09:47.187606 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.175362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.1.154.188:42293)
0620 19:09:47.175858 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.175870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.175884 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:09:47.175944 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.175949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.175993 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.176092 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.176787 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.177803 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.179253 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.179739 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.179856 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.183367 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:47.183373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:47.183377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:47.183411 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:47.185711 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.186251 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.186255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.186260 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.186364 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.186657 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.186661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.186662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.186911 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:47.187004 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:47.187323 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 19:09:47.187449 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":167,"threads_started":1}
I20260620 19:09:47.187758 24033 log.cc:824] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:47.190435 24033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f: No bootstrap required, opened a new log
I20260620 19:09:47.191171 23958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab: Bootstrap starting.
I20260620 19:09:47.194764 24033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } }
I20260620 19:09:47.195015 24033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea65089539a4cf39b5e969c013db61f, State: Initialized, Role: FOLLOWER
I20260620 19:09:47.195232 23958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:47.195536 24033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } }
I20260620 19:09:47.196133 23958 log.cc:824] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:47.196661 24130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } } }
I20260620 19:09:47.196815 24130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:09:47.197116 24033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:47.198446 23958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab: No bootstrap required, opened a new log
I20260620 19:09:47.201884 23958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } }
I20260620 19:09:47.202118 23958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 480d2acde73040ec9deaedad1fcb91ab, State: Initialized, Role: FOLLOWER
I20260620 19:09:47.202455 23958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } }
I20260620 19:09:47.203011 24143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } } }
I20260620 19:09:47.203213 24143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:09:47.203742 23958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:47.203816 23932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:09:47.203930 23932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:47.201210 23965 master_runner.cc:186] Error getting master registration for 127.1.154.190:39143: OK, instance_id { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" instance_seqno: 1781982586532190 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:09:47.204802 23965 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:09:47.205238 23932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } }
I20260620 19:09:47.207208 24007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "480d2acde73040ec9deaedad1fcb91ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0ea65089539a4cf39b5e969c013db61f"
I20260620 19:09:47.207439 24007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:09:47.207937 24142 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:09:47.208094 24142 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:09:47.208793 24007 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 480d2acde73040ec9deaedad1fcb91ab in term 1.
I20260620 19:09:47.209237 24083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "480d2acde73040ec9deaedad1fcb91ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f10f11b001b64121920449e87e5c8463"
I20260620 19:09:47.209491 24083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:47.210789 24083 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 480d2acde73040ec9deaedad1fcb91ab in term 1.
I20260620 19:09:47.212675 23908 leader_election.cc:302] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 480d2acde73040ec9deaedad1fcb91ab, f10f11b001b64121920449e87e5c8463; no voters:
W20260620 19:09:47.215533 24155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:09:47.215619 24155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:09:47.217901 23890 master_runner.cc:263] No leader master found from master 480d2acde73040ec9deaedad1fcb91ab
I20260620 19:09:47.217973 23890 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:09:47.206218 23932 leader_election.cc:288] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [CANDIDATE]: Term 1 election: Requested vote from peers 0ea65089539a4cf39b5e969c013db61f (127.1.154.189:34069), f10f11b001b64121920449e87e5c8463 (127.1.154.188:42293)
I20260620 19:09:47.220170 24143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:47.220430 24143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [term 1 LEADER]: Becoming Leader. State: Replica: 480d2acde73040ec9deaedad1fcb91ab, State: Running, Role: LEADER
I20260620 19:09:47.220788 24143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } }
I20260620 19:09:47.222741 24143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 480d2acde73040ec9deaedad1fcb91ab. Latest consensus state: current_term: 1 leader_uuid: "480d2acde73040ec9deaedad1fcb91ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } } }
I20260620 19:09:47.222860 24143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:47.227180 24163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:47.228252 24163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:47.240113 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.224275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.1.154.188:42293)
0620 19:09:47.224390 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.224394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.229980 (+ 5586us) server_negotiation.cc:407] Connection header received
0620 19:09:47.230177 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.230181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.230234 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.230311 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.231447 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.232249 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.234285 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.234676 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.234767 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.236588 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:47.236592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:47.236595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:47.236619 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:47.238362 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.238810 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.238814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.238816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.238904 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.239169 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.239172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.239174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.239376 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:47.239487 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:47.239816 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 19:09:47.239943 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:09:47.243217 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.224924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.1.154.190:39143)
0620 19:09:47.225040 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.225044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.233825 (+ 8781us) server_negotiation.cc:407] Connection header received
0620 19:09:47.233992 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.233995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.234047 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.234096 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.235025 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.236125 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.237542 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.237901 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.237973 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.239563 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:47.239567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:47.239570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:47.239595 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:47.241251 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.241781 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.241784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.241786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.241872 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.242189 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.242204 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.242207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.242452 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:47.242591 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:47.242959 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 19:09:47.243069 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:09:47.246438 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:47.223592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.1.154.189:34069)
0620 19:09:47.223714 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.223719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.227464 (+ 3745us) server_negotiation.cc:407] Connection header received
0620 19:09:47.227659 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.227663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.227713 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.227767 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.230415 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.231202 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.240542 (+ 9340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:47.240933 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:47.241027 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:47.242736 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:47.242740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:47.242743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:47.242772 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:47.244642 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.245119 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.245123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.245125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.245203 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:47.245476 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:47.245479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:47.245481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:47.245731 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:47.245844 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:47.246176 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 19:09:47.246288 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:09:47.251039 24083 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463 [term 1 FOLLOWER]: Refusing update from remote peer 480d2acde73040ec9deaedad1fcb91ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:47.251122 24007 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f [term 1 FOLLOWER]: Refusing update from remote peer 480d2acde73040ec9deaedad1fcb91ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:47.251837 24143 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:47.252377 24143 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:09:47.255780 23986 tablet.cc:2369] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:09:47.256435 24130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 480d2acde73040ec9deaedad1fcb91ab. Latest consensus state: current_term: 1 leader_uuid: "480d2acde73040ec9deaedad1fcb91ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } } }
I20260620 19:09:47.256556 24130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:09:47.256505 24115 sys_catalog.cc:455] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 480d2acde73040ec9deaedad1fcb91ab. Latest consensus state: current_term: 1 leader_uuid: "480d2acde73040ec9deaedad1fcb91ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } } }
I20260620 19:09:47.256629 24115 sys_catalog.cc:458] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:09:47.267442 24062 tablet.cc:2369] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:09:47.269562 24161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "480d2acde73040ec9deaedad1fcb91ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } } }
I20260620 19:09:47.269733 24161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:47.271170 24130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "480d2acde73040ec9deaedad1fcb91ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } } }
I20260620 19:09:47.271289 24130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:09:47.271672 24167 mvcc.cc:205] Tried to move back new op lower bound from 7299000677349879808 to 7299000677259677696. Current Snapshot: MvccSnapshot[applied={T|T < 7299000677349879808}]
I20260620 19:09:47.271818 24168 mvcc.cc:205] Tried to move back new op lower bound from 7299000677349879808 to 7299000677259677696. Current Snapshot: MvccSnapshot[applied={T|T < 7299000677349879808}]
I20260620 19:09:47.272780 24143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "480d2acde73040ec9deaedad1fcb91ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } } }
I20260620 19:09:47.272892 24143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:47.273947 24115 sys_catalog.cc:455] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "480d2acde73040ec9deaedad1fcb91ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480d2acde73040ec9deaedad1fcb91ab" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 39143 } } peers { permanent_uuid: "0ea65089539a4cf39b5e969c013db61f" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 34069 } } peers { permanent_uuid: "f10f11b001b64121920449e87e5c8463" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 42293 } } }
I20260620 19:09:47.274062 24115 sys_catalog.cc:458] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:09:47.274400 24169 mvcc.cc:205] Tried to move back new op lower bound from 7299000677349879808 to 7299000677259677696. Current Snapshot: MvccSnapshot[applied={T|T < 7299000677349879808}]
I20260620 19:09:47.276324 24163 catalog_manager.cc:1347] Generated new cluster ID: 15e7f3280f514def946491d5b85490b1
I20260620 19:09:47.276393 24163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:47.311053 24163 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:47.312393 24163 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:47.321655 24163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 480d2acde73040ec9deaedad1fcb91ab: Generated new TSK 0
I20260620 19:09:47.323690 24163 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:09:47.357453 23961 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:47.355022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46818 (local address 127.1.154.190:39143)
0620 19:09:47.355171 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.355177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.356258 (+ 1081us) server_negotiation.cc:407] Connection header received
0620 19:09:47.356408 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.356412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.356470 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.356570 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.357296 (+ 726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46818: BlockingRecv error: recv got EOF from 127.0.0.1:46818 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 19:09:47.358860 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:47.356102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.1.154.189:34069)
0620 19:09:47.356259 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.356263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.357910 (+ 1647us) server_negotiation.cc:407] Connection header received
0620 19:09:47.358091 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.358095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.358143 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.358196 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.358728 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51952: BlockingRecv error: recv got EOF from 127.0.0.1:51952 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 19:09:47.359339 24128 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:47.356956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45572 (local address 127.1.154.188:42293)
0620 19:09:47.357069 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.357074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.358059 (+ 985us) server_negotiation.cc:407] Connection header received
0620 19:09:47.358249 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.358252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.358303 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.358361 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.359141 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45572: BlockingRecv error: recv got EOF from 127.0.0.1:45572 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 19:09:47.363322 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:47.362003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.1.154.189:34069)
0620 19:09:47.362107 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.362111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.362534 (+ 423us) server_negotiation.cc:407] Connection header received
0620 19:09:47.362656 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.362660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.362733 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.362821 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.363222 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51958: BlockingRecv error: recv got EOF from 127.0.0.1:51958 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 19:09:47.367324 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:47.366521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.1.154.189:34069)
0620 19:09:47.366615 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.366619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.366631 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:09:47.366675 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.366678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.366721 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.366808 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.367233 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51972: BlockingRecv error: recv got EOF from 127.0.0.1:51972 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 19:09:47.372499 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:47.371228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.1.154.189:34069)
0620 19:09:47.371333 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.371337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.371586 (+ 249us) server_negotiation.cc:407] Connection header received
0620 19:09:47.371787 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.371790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.371837 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.371888 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.372388 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51982: BlockingRecv error: recv got EOF from 127.0.0.1:51982 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 19:09:47.380071 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:47.378874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51992 (local address 127.1.154.189:34069)
0620 19:09:47.378984 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.378988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.379218 (+ 230us) server_negotiation.cc:407] Connection header received
0620 19:09:47.379343 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.379347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.379399 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.379487 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.379963 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51992: BlockingRecv error: recv got EOF from 127.0.0.1:51992 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 19:09:47.391307 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:47.390105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.1.154.189:34069)
0620 19:09:47.390222 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.390226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.390478 (+ 252us) server_negotiation.cc:407] Connection header received
0620 19:09:47.390608 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.390612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.390663 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.390745 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.391194 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52004: BlockingRecv error: recv got EOF from 127.0.0.1:52004 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 19:09:47.411088 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:47.409652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52020 (local address 127.1.154.189:34069)
0620 19:09:47.409784 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.409789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.410104 (+ 315us) server_negotiation.cc:407] Connection header received
0620 19:09:47.410253 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.410256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.410315 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.410429 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.410972 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52020: BlockingRecv error: recv got EOF from 127.0.0.1:52020 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 19:09:47.446628 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:47.445397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.1.154.189:34069)
0620 19:09:47.445552 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.445556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.445728 (+ 172us) server_negotiation.cc:407] Connection header received
0620 19:09:47.445911 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.445914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.445969 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.446064 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.446522 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52022: BlockingRecv error: recv got EOF from 127.0.0.1:52022 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 19:09:47.514189 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:47.512908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.1.154.189:34069)
0620 19:09:47.513090 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.513094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.513174 (+ 80us) server_negotiation.cc:407] Connection header received
0620 19:09:47.513323 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.513325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.513375 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.513490 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.514036 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52038: BlockingRecv error: recv got EOF from 127.0.0.1:52038 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 19:09:47.645767 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:47.644373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52046 (local address 127.1.154.189:34069)
0620 19:09:47.644574 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.644580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.644746 (+ 166us) server_negotiation.cc:407] Connection header received
0620 19:09:47.644957 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.644960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.645022 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.645125 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.645648 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52046: BlockingRecv error: recv got EOF from 127.0.0.1:52046 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260620 19:09:47.905292 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:47.904084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.1.154.189:34069)
0620 19:09:47.904236 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:47.904239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:47.904384 (+ 145us) server_negotiation.cc:407] Connection header received
0620 19:09:47.904609 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:47.904612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:47.904668 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:47.904751 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:47.905161 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52056: BlockingRecv error: recv got EOF from 127.0.0.1:52056 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 19:09:48.093210 24126 catalog_manager.cc:1259] Loaded cluster ID: 15e7f3280f514def946491d5b85490b1
I20260620 19:09:48.093307 24126 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463: loading cluster ID for follower catalog manager: success
I20260620 19:09:48.095062 24126 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463: acquiring CA information for follower catalog manager: success
I20260620 19:09:48.096196 24126 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f10f11b001b64121920449e87e5c8463: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:09:48.209501 24142 catalog_manager.cc:1259] Loaded cluster ID: 15e7f3280f514def946491d5b85490b1
I20260620 19:09:48.209740 24142 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f: loading cluster ID for follower catalog manager: success
I20260620 19:09:48.211753 24142 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f: acquiring CA information for follower catalog manager: success
I20260620 19:09:48.212896 24142 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ea65089539a4cf39b5e969c013db61f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:09:48.424726 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:48.419962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52060 (local address 127.1.154.189:34069)
0620 19:09:48.420248 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:48.420254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:48.420272 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:09:48.420364 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:48.420368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:48.420509 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:48.420640 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:48.420915 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:48.422090 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:48.423056 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:48.423317 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:48.423437 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:48.423780 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:48.424266 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:48.424410 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:48.424482 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 19:09:48.424545 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
I20260620 19:09:48.432423 24246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:48.428407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.1.154.188:42293)
0620 19:09:48.428716 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:48.428720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:48.428734 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:09:48.428816 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:48.428818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:48.428934 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:48.429078 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:48.429310 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:48.429909 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:48.430839 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:48.431094 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:48.431217 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:48.431537 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:48.431972 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:48.432116 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:48.432185 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 19:09:48.432242 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":94,"threads_started":1}
I20260620 19:09:48.440675 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:48.435766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.1.154.189:34069)
0620 19:09:48.435918 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:48.435922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:48.436202 (+ 280us) server_negotiation.cc:407] Connection header received
0620 19:09:48.436418 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:48.436423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:48.436489 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:48.436546 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:48.437006 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:48.437815 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:48.439463 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:48.439711 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:48.439785 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:48.440149 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:48.440242 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:48.440321 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:48.440472 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 19:09:48.440539 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:09:48.440747 24246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:48.436324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.1.154.188:42293)
0620 19:09:48.436426 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:48.436429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:48.436713 (+ 284us) server_negotiation.cc:407] Connection header received
0620 19:09:48.436836 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:48.436840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:48.436889 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:48.436966 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:48.438013 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:48.438515 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:48.439530 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:48.439790 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:48.439902 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:48.440218 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:48.440305 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:48.440378 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:48.440530 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 19:09:48.440588 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 19:09:48.441661 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:48.435703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.1.154.190:39143)
0620 19:09:48.436071 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:48.436075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:48.436091 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:09:48.436311 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:48.436315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:48.436615 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:48.437024 (+ 409us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:48.437125 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:48.437922 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:48.439841 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:48.440234 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:48.440342 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:48.440652 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:48.441005 (+ 353us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:48.441377 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:48.441413 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 19:09:48.441471 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":97,"threads_started":1}
I20260620 19:09:48.445789 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 23890
I20260620 19:09:48.452524 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 23965
I20260620 19:09:48.458753 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 24041
2026-06-20T19:09:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3296 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:48 dist-test-slave-6wrf krb5kdc[24257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:48 dist-test-slave-6wrf krb5kdc[24257](info): set up 2 sockets
Jun 20 19:09:48 dist-test-slave-6wrf krb5kdc[24257](info): commencing operation
krb5kdc: starting...
W20260620 19:09:49.546121 1642 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/test-user.keytab.
Jun 20 19:09:49 dist-test-slave-6wrf krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.154.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
Entry for principal kudu/127.1.154.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab.
I20260620 19:09:49.682087 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43911
--webserver_interface=127.1.154.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:43911,127.1.154.189:40481,127.1.154.188:41727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:49.794660 24270 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:09:49.794984 24270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:49.795011 24270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:49.795112 24270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:49.799188 24270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:49.799263 24270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:49.799285 24270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:49.799305 24270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:49.799371 24270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:49.804755 24270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:43911,127.1.154.189:40481,127.1.154.188:41727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.190.keytab
--principal=kudu/127.1.154.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.190:43911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.154.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:49.805989 24270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:49.807490 24270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:49.817036 24278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:49.817032 24276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:49.817143 24275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:49.817639 24270 server_base.cc:1034] running on GCE node
I20260620 19:09:49.818528 24270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:49.820184 24270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:49.821441 24270 hybrid_clock.cc:648] HybridClock initialized: now 1781982589821400 us; error 56 us; skew 500 ppm
Jun 20 19:09:49 dist-test-slave-6wrf krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982589, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:49.824702 24270 init.cc:377] Logged in from keytab as kudu/127.1.154.190@KRBTEST.COM (short username kudu)
I20260620 19:09:49.825817 24270 webserver.cc:460] Webserver started at http://127.1.154.190:40255/ using document root <none> and password file <none>
I20260620 19:09:49.826449 24270 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:49.826555 24270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:49.826805 24270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:49.829075 24270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/data/instance:
uuid: "5ce5fa559e194da49f04e38f096dc5a0"
format_stamp: "Formatted at 2026-06-20 19:09:49 on dist-test-slave-6wrf"
server_key: "14354de6014334cd7b17b6883f5aac9e"
server_key_iv: "371eb2415b0a28f4bc50a2ce64ff8b24"
server_key_version: "encryptionkey@0"
I20260620 19:09:49.829777 24270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/wal/instance:
uuid: "5ce5fa559e194da49f04e38f096dc5a0"
format_stamp: "Formatted at 2026-06-20 19:09:49 on dist-test-slave-6wrf"
server_key: "14354de6014334cd7b17b6883f5aac9e"
server_key_iv: "371eb2415b0a28f4bc50a2ce64ff8b24"
server_key_version: "encryptionkey@0"
I20260620 19:09:49.833755 24270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:09:49.836207 24285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:49.837549 24270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:09:49.837738 24270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/wal
uuid: "5ce5fa559e194da49f04e38f096dc5a0"
format_stamp: "Formatted at 2026-06-20 19:09:49 on dist-test-slave-6wrf"
server_key: "14354de6014334cd7b17b6883f5aac9e"
server_key_iv: "371eb2415b0a28f4bc50a2ce64ff8b24"
server_key_version: "encryptionkey@0"
I20260620 19:09:49.837868 24270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:49.863018 24270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:49.866417 24270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:49.866660 24270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:49.874150 24270 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.190:43911
I20260620 19:09:49.874444 24337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.190:43911 every 8 connection(s)
I20260620 19:09:49.875207 24270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:49.877964 24338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:49.881817 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 24270
I20260620 19:09:49.881467 24338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:09:49.881948 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-0/wal/instance
I20260620 19:09:49.882233 1642 external_mini_cluster.cc:1428] Setting key 3e1f67cc2b691ee7513d9ca2157086b4
WARNING: no policy specified for kudu/127.1.154.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.189@KRBTEST.COM" created.
Jun 20 19:09:49 dist-test-slave-6wrf krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982589, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:49.900985 24342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:49.885142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45240 (local address 127.1.154.190:43911)
0620 19:09:49.885426 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:49.885435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:49.885469 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:09:49.886130 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:49.886134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:49.886320 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:49.886459 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:49.887568 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:49.888440 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:49.889960 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:49.890349 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:49.890556 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:49.895509 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:49.895520 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:49.895539 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:49.895577 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:49.897429 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:49.898163 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:49.898167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:49.898168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:49.898368 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:49.898805 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:49.898807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:49.898809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:49.899220 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:49.899328 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:49.899891 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 19:09:49.900131 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
I20260620 19:09:49.900992 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:49.884196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43911 (local address 127.0.0.1:45240)
0620 19:09:49.884693 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:49.884734 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:49.885842 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:49.886522 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:49.886530 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:49.887039 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:49.887399 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:49.887420 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:49.888573 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:49.888576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:49.889823 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:49.889833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:49.890556 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:49.890560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:49.890657 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:49.891287 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:49.891304 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:49.895279 (+ 3975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:49.897573 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:49.897587 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:49.897632 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:49.897999 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:49.898485 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:49.898488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:49.898492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:49.898661 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:49.899363 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:49.899374 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:49.899571 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 19:09:49.899791 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 19:09:49.900227 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":61,"thread_start_us":145,"threads_started":1}
I20260620 19:09:49.904206 24338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:09:49.905066 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:49.904761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:40481 (local address 127.0.0.1:51280)
0620 19:09:49.904920 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:49.905003 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 19:09:49.905296 24288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.154.189:40481: connect: Connection refused (error 111)
I20260620 19:09:49.906270 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:49.905958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:40481 (local address 127.0.0.1:51294)
0620 19:09:49.906142 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:49.906206 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 19:09:49.906569 24338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.189:40481: Network error: Client connection negotiation failed: client connection to 127.1.154.189:40481: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.189.keytab.
Entry for principal kudu/127.1.154.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.189.keytab.
I20260620 19:09:49.912894 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.189:40481
--webserver_interface=127.1.154.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.189.keytab
--principal=kudu/127.1.154.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:43911,127.1.154.189:40481,127.1.154.188:41727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:09:49.955991 24338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } attempt: 1
I20260620 19:09:49.957034 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:49.956619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:40481 (local address 127.0.0.1:51296)
0620 19:09:49.956799 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:49.956912 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 19:09:49.957931 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:49.957695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:40481 (local address 127.0.0.1:51298)
0620 19:09:49.957806 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:49.957861 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 19:09:49.958096 24338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.189:40481: Network error: Client connection negotiation failed: client connection to 127.1.154.189:40481: connect: Connection refused (error 111)
I20260620 19:09:50.026305 24338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } attempt: 2
I20260620 19:09:50.027204 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.026869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:40481 (local address 127.0.0.1:51304)
0620 19:09:50.027048 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.027118 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 19:09:50.027669 24345 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:09:50.028024 24345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:50.028136 24345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:50.028208 24345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 19:09:50.028206 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.027920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:40481 (local address 127.0.0.1:51314)
0620 19:09:50.028067 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.028136 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.189:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 19:09:50.028371 24338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.189:40481: Network error: Client connection negotiation failed: client connection to 127.1.154.189:40481: connect: Connection refused (error 111)
W20260620 19:09:50.032970 24345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:50.033068 24345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:50.033124 24345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:50.033164 24345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:50.033200 24345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:50.039260 24345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:43911,127.1.154.189:40481,127.1.154.188:41727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.189.keytab
--principal=kudu/127.1.154.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.189:40481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.154.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:50.040459 24345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:50.041725 24345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:50.051069 24350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:50.051052 24351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:50.051055 24353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:50.051380 24345 server_base.cc:1034] running on GCE node
I20260620 19:09:50.052538 24345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:50.053891 24345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:50.055094 24345 hybrid_clock.cc:648] HybridClock initialized: now 1781982590055047 us; error 65 us; skew 500 ppm
Jun 20 19:09:50 dist-test-slave-6wrf krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982590, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:50.057889 24345 init.cc:377] Logged in from keytab as kudu/127.1.154.189@KRBTEST.COM (short username kudu)
I20260620 19:09:50.058774 24345 webserver.cc:460] Webserver started at http://127.1.154.189:39381/ using document root <none> and password file <none>
I20260620 19:09:50.059378 24345 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:50.059453 24345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:50.059666 24345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:50.061699 24345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/data/instance:
uuid: "d95caded10694b32a38ea1468ab842b8"
format_stamp: "Formatted at 2026-06-20 19:09:50 on dist-test-slave-6wrf"
server_key: "fc815e9ec101581c5d17f79d29e98650"
server_key_iv: "2da5747c5ba4490d2095014bdf0bcf05"
server_key_version: "encryptionkey@0"
I20260620 19:09:50.062296 24345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/wal/instance:
uuid: "d95caded10694b32a38ea1468ab842b8"
format_stamp: "Formatted at 2026-06-20 19:09:50 on dist-test-slave-6wrf"
server_key: "fc815e9ec101581c5d17f79d29e98650"
server_key_iv: "2da5747c5ba4490d2095014bdf0bcf05"
server_key_version: "encryptionkey@0"
I20260620 19:09:50.065862 24345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:09:50.068586 24360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:50.070628 24345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:50.070750 24345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/wal
uuid: "d95caded10694b32a38ea1468ab842b8"
format_stamp: "Formatted at 2026-06-20 19:09:50 on dist-test-slave-6wrf"
server_key: "fc815e9ec101581c5d17f79d29e98650"
server_key_iv: "2da5747c5ba4490d2095014bdf0bcf05"
server_key_version: "encryptionkey@0"
I20260620 19:09:50.070852 24345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:50.084895 24345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:50.088368 24345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:50.088549 24345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:50.096493 24345 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.189:40481
I20260620 19:09:50.096791 24412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.189:40481 every 8 connection(s)
I20260620 19:09:50.097685 24345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/data/info.pb
I20260620 19:09:50.100232 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 24345
I20260620 19:09:50.100353 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-1/wal/instance
I20260620 19:09:50.100668 1642 external_mini_cluster.cc:1428] Setting key d6ab74b4eb2b7236773dddb703c3ac7a
I20260620 19:09:50.100730 24413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:50.104764 24413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.154.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.154.188@KRBTEST.COM" created.
Jun 20 19:09:50 dist-test-slave-6wrf krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982590, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.189@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:50.119572 24338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } attempt: 3
I20260620 19:09:50.120712 24342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.106755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45256 (local address 127.1.154.190:43911)
0620 19:09:50.106948 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.106964 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.107705 (+ 741us) server_negotiation.cc:407] Connection header received
0620 19:09:50.108671 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.108676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.108746 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.108872 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.110163 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.110693 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.111839 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.112146 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.112264 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.115707 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:50.115714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:50.115717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:50.115752 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:50.118151 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.118853 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.118856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.118857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.118934 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.119383 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.119386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.119387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.119592 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:50.119728 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:50.120409 (+ 681us) server_negotiation.cc:299] Negotiation successful
0620 19:09:50.120547 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 19:09:50.121510 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.107072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43911 (local address 127.0.0.1:45256)
0620 19:09:50.107543 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.107592 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:50.108457 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:50.109072 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:50.109079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:50.109426 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:50.109965 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.109978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.110865 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.110868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.111709 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.111718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.112300 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.112304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.112496 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.113152 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:50.113179 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:50.115518 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:50.118278 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.118286 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.118306 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.118723 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.119035 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.119040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.119044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.119242 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.119737 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:50.119747 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:50.120118 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 19:09:50.120344 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 19:09:50.120682 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":150,"threads_started":1}
I20260620 19:09:50.123021 24413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:09:50 dist-test-slave-6wrf krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982589, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for kudu/127.1.154.189@KRBTEST.COM
Jun 20 19:09:50 dist-test-slave-6wrf krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982590, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.189@KRBTEST.COM for kudu/127.1.154.189@KRBTEST.COM
I20260620 19:09:50.132262 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.120037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:40481 (local address 127.0.0.1:51318)
0620 19:09:50.120226 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.120242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:50.120326 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:50.121689 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:50.121693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:50.121869 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:50.122008 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.122016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.123292 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.123295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.124197 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.124209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.124901 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.124905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.125028 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.125538 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:50.125553 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:50.127201 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:50.130066 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.130070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.130072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.130404 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.130873 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.130878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.130883 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.130993 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.131656 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:50.131660 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:50.131776 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 19:09:50.131909 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 19:09:50.132102 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 19:09:50.132467 24418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.120486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.1.154.189:40481)
0620 19:09:50.121216 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.121229 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.121253 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:09:50.121292 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.121299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.121499 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.122188 (+ 689us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.122298 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.123162 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.124325 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.124781 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.124867 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.127466 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:50.127485 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:50.127502 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:50.127593 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:50.129876 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.130590 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.130601 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.130608 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.130739 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.131101 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.131104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.131106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.131481 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:50.131691 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:50.132054 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 19:09:50.132239 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.154.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.188.keytab.
Entry for principal kudu/127.1.154.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.188.keytab.
I20260620 19:09:50.133348 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.123498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:40481 (local address 127.0.0.1:51326)
0620 19:09:50.123617 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.123632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:50.123715 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:50.123973 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:50.123975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:50.124142 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:50.124249 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.124258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.125058 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.125062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.125584 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.125591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.126363 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.126367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.126477 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.127005 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:50.127019 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:50.129138 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:50.131772 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.131774 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.131776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.132025 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.132327 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.132333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.132337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.132447 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.132973 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:50.132975 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:50.133072 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 19:09:50.133155 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 19:09:50.133250 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 19:09:50.133586 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.123523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.1.154.189:40481)
0620 19:09:50.123770 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.123776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.123787 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:09:50.123821 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.123824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.123854 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.123922 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.124358 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.124949 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.125771 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.126209 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.126291 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.129319 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:50.129323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:50.129326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:50.129351 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:50.131691 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.132128 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.132131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.132133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.132216 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.132627 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.132633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.132637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.132848 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:50.132937 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:50.133248 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 19:09:50.133374 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":96,"threads_started":1}
I20260620 19:09:50.133634 1642 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8BnTdh/build/debug/bin/kudu
/tmp/dist-test-task8BnTdh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.188:41727
--webserver_interface=127.1.154.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.188.keytab
--principal=kudu/127.1.154.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.154.148:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.154.190:43911,127.1.154.189:40481,127.1.154.188:41727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:09:50.136508 24338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:09:50.136554 24413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:09:50.137228 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.136974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:41727 (local address 127.0.0.1:60224)
0620 19:09:50.137098 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.137169 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 19:09:50.137315 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.136998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:41727 (local address 127.0.0.1:60214)
0620 19:09:50.137174 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.137231 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260620 19:09:50.137467 24363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
I20260620 19:09:50.138319 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.138107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:41727 (local address 127.0.0.1:60234)
0620 19:09:50.138211 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.138260 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 19:09:50.138329 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.137788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:41727 (local address 127.0.0.1:60226)
0620 19:09:50.138166 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.138226 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":115,"threads_started":1}
W20260620 19:09:50.138489 24338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:41727: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
W20260620 19:09:50.138512 24413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:41727: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
I20260620 19:09:50.169689 24338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } attempt: 1
I20260620 19:09:50.170635 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.170257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:41727 (local address 127.0.0.1:60240)
0620 19:09:50.170459 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.170538 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 19:09:50.171478 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.171221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:41727 (local address 127.0.0.1:60242)
0620 19:09:50.171338 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.171395 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 19:09:50.171677 24338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:41727: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
I20260620 19:09:50.187800 24413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } attempt: 1
I20260620 19:09:50.188894 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.188494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:41727 (local address 127.0.0.1:60244)
0620 19:09:50.188711 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.188794 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 19:09:50.190937 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.190697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:41727 (local address 127.0.0.1:60256)
0620 19:09:50.190830 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.190880 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 19:09:50.191228 24413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:41727: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
I20260620 19:09:50.246910 24338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } attempt: 2
I20260620 19:09:50.247769 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.247442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:41727 (local address 127.0.0.1:60260)
0620 19:09:50.247625 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.247691 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 19:09:50.248538 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.248337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:41727 (local address 127.0.0.1:60268)
0620 19:09:50.248444 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.248487 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 19:09:50.248404 24420 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:09:50.248694 24338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:41727: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
W20260620 19:09:50.248725 24420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:50.248768 24420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:50.248835 24420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:50.252965 24420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:50.253017 24420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:50.253037 24420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:50.253054 24420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:50.253099 24420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:50.258430 24420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.154.148:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.154.190:43911,127.1.154.189:40481,127.1.154.188:41727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/krb5kdc/kudu_127.1.154.188.keytab
--principal=kudu/127.1.154.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.154.188:41727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.154.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:50.259410 24413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } attempt: 2
I20260620 19:09:50.259543 24420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:50.260411 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.260078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:41727 (local address 127.0.0.1:60274)
0620 19:09:50.260277 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.260337 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 19:09:50.260771 24420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:09:50.261231 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.261030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:41727 (local address 127.0.0.1:60278)
0620 19:09:50.261134 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.261179 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 19:09:50.261469 24413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.154.188:41727: Network error: Client connection negotiation failed: client connection to 127.1.154.188:41727: connect: Connection refused (error 111)
W20260620 19:09:50.269856 24427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:50.269888 24428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:50.269906 24430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:50.269979 24420 server_base.cc:1034] running on GCE node
I20260620 19:09:50.270979 24420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:50.272461 24420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:50.273720 24420 hybrid_clock.cc:648] HybridClock initialized: now 1781982590273674 us; error 56 us; skew 500 ppm
Jun 20 19:09:50 dist-test-slave-6wrf krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982590, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:50.276991 24420 init.cc:377] Logged in from keytab as kudu/127.1.154.188@KRBTEST.COM (short username kudu)
I20260620 19:09:50.278055 24420 webserver.cc:460] Webserver started at http://127.1.154.188:41631/ using document root <none> and password file <none>
I20260620 19:09:50.278628 24420 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:50.278697 24420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:50.278916 24420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:50.281059 24420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/data/instance:
uuid: "20069f85bdae4fe68a230e792a365db5"
format_stamp: "Formatted at 2026-06-20 19:09:50 on dist-test-slave-6wrf"
server_key: "371a62d6f35664eba04beddf552664b3"
server_key_iv: "8fbbdef637e2dfcd53c1c779c244ff13"
server_key_version: "encryptionkey@0"
I20260620 19:09:50.281682 24420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/wal/instance:
uuid: "20069f85bdae4fe68a230e792a365db5"
format_stamp: "Formatted at 2026-06-20 19:09:50 on dist-test-slave-6wrf"
server_key: "371a62d6f35664eba04beddf552664b3"
server_key_iv: "8fbbdef637e2dfcd53c1c779c244ff13"
server_key_version: "encryptionkey@0"
I20260620 19:09:50.285327 24420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:50.287753 24437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:50.289131 24420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:09:50.289229 24420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/data,/tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/wal
uuid: "20069f85bdae4fe68a230e792a365db5"
format_stamp: "Formatted at 2026-06-20 19:09:50 on dist-test-slave-6wrf"
server_key: "371a62d6f35664eba04beddf552664b3"
server_key_iv: "8fbbdef637e2dfcd53c1c779c244ff13"
server_key_version: "encryptionkey@0"
I20260620 19:09:50.289348 24420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:50.305015 24420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:50.308929 24420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:50.309141 24420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:50.317031 24420 rpc_server.cc:307] RPC server started. Bound to: 127.1.154.188:41727
I20260620 19:09:50.317350 24489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.154.188:41727 every 8 connection(s)
I20260620 19:09:50.318048 24420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/data/info.pb
I20260620 19:09:50.321412 1642 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8BnTdh/build/debug/bin/kudu as pid 24420
I20260620 19:09:50.321557 1642 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8BnTdh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982293404112-1642-0/minicluster-data/master-2/wal/instance
I20260620 19:09:50.321888 24490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:50.321978 1642 external_mini_cluster.cc:1428] Setting key 1d3048fcd97c4ec18a61c7f57f0c4e99
I20260620 19:09:50.328147 24490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:09:50 dist-test-slave-6wrf krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.188@KRBTEST.COM
Jun 20 19:09:50 dist-test-slave-6wrf krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982590, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.188@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:50.340361 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.323700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60280 (local address 127.1.154.188:41727)
0620 19:09:50.324164 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.324179 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.324215 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:09:50.324898 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.324913 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.325261 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.325539 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.327244 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.328902 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.330092 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.330736 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.330904 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.334453 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:50.334461 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:50.334473 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:50.334512 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:50.337328 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.337916 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.337928 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.337932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.338035 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.338338 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.338346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.338347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.338730 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:50.338946 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:50.339222 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 19:09:50.339505 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":172,"threads_started":1}
I20260620 19:09:50.341339 24342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.330491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45264 (local address 127.1.154.190:43911)
0620 19:09:50.330695 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.330701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.331121 (+ 420us) server_negotiation.cc:407] Connection header received
0620 19:09:50.331291 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.331296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.331350 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.331465 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.332851 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.333364 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.334687 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.334906 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.335041 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.337914 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:50.337917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:50.337919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:50.337952 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:50.339420 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.339891 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.339897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.339899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.339989 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.340274 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.340279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.340280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.340549 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:50.340674 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:50.341052 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 19:09:50.341183 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 19:09:50.341351 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.330367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.190:43911 (local address 127.0.0.1:45264)
0620 19:09:50.330972 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.331024 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:50.331180 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:50.331576 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:50.331583 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:50.332326 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:50.332630 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.332642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.333522 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.333526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.334537 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.334548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.335036 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.335042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.335199 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.335730 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:50.335752 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:50.337767 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:50.339529 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.339537 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.339542 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.339783 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.340085 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.340089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.340091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.340183 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.340702 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:50.340709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:50.340831 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 19:09:50.340988 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 19:09:50.341174 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":303,"threads_started":1}
I20260620 19:09:50.343037 24490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:09:50.347887 24338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } attempt: 3
Jun 20 19:09:50 dist-test-slave-6wrf krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982590, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.188@KRBTEST.COM for kudu/127.1.154.189@KRBTEST.COM
Jun 20 19:09:50 dist-test-slave-6wrf krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.190@KRBTEST.COM
I20260620 19:09:50.352666 24413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } attempt: 3
Jun 20 19:09:50 dist-test-slave-6wrf krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982589, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.190@KRBTEST.COM for kudu/127.1.154.188@KRBTEST.COM
I20260620 19:09:50.353519 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.343697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.1.154.189:40481)
0620 19:09:50.343838 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.343843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.343860 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:09:50.343907 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.343910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.343957 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.344044 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.344553 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.345214 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.346536 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.346879 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.346989 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.349881 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:50.349884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:50.349887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:50.349911 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:50.351590 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.352064 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.352070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.352075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.352162 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.352514 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.352518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.352520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.352762 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:50.352865 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:50.353205 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 19:09:50.353324 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:09:50.353991 24342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.342683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.1.154.190:43911)
0620 19:09:50.342786 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.342789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.342852 (+ 63us) server_negotiation.cc:407] Connection header received
0620 19:09:50.342973 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.342976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.343018 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.343100 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.344174 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.344804 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.345895 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.346543 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.346641 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.350124 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:50.350127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:50.350130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:50.350160 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:50.351681 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.352148 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.352151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.352153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.352236 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.352550 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.352554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.352556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.352831 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:50.353144 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:50.353728 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 19:09:50.353858 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:09:50.354030 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.343595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.189:40481 (local address 127.0.0.1:51332)
0620 19:09:50.343711 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.343726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:50.343796 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:50.344085 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:50.344087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:50.344291 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:50.344409 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.344415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.345322 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.345325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.346408 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.346419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.347096 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.347103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.347345 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.347808 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:50.347822 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:50.349734 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:50.351701 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.351706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.351711 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.351930 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.352304 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.352310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.352312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.352406 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.352914 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:50.352917 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:50.353006 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 19:09:50.353735 (+ 729us) client_negotiation.cc:240] Negotiation successful
0620 19:09:50.353881 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 19:09:50.355950 24490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:09:50.358675 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.348341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:41727 (local address 127.0.0.1:60294)
0620 19:09:50.348604 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.348615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:50.348703 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:50.348966 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:50.348969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:50.349127 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:50.349196 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.349201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.350138 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.350144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.350992 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.351007 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.351702 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.351705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.351793 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.352244 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:50.352259 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:50.353737 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:50.357078 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.357082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.357084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.357339 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.357667 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.357670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.357674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.357763 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.358204 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:50.358207 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:50.358294 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 19:09:50.358386 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 19:09:50.358505 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"spinlock_wait_cycles":1280}
I20260620 19:09:50.359211 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.348627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.1.154.188:41727)
0620 19:09:50.348765 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.348772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.348785 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:09:50.348816 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.348818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.348859 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.348943 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.349308 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.350000 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.351150 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.351557 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.352009 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.354018 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:50.354021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:50.354024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:50.354048 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:50.356951 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.357445 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.357450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.357451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.357537 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.357885 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.357889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.357891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.358073 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:50.358161 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:50.358965 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 19:09:50.359073 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 19:09:50 dist-test-slave-6wrf krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982590, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.189@KRBTEST.COM for kudu/127.1.154.188@KRBTEST.COM
Jun 20 19:09:50 dist-test-slave-6wrf krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982590, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.154.188@KRBTEST.COM for kudu/127.1.154.188@KRBTEST.COM
I20260620 19:09:50.362870 24338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0: Bootstrap starting.
I20260620 19:09:50.365933 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.356431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:41727 (local address 127.0.0.1:60304)
0620 19:09:50.356690 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.356703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:50.356780 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:50.357517 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:50.357520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:50.357700 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:50.357786 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.357796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.358748 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.358753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.359469 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.359477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.360291 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.360294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.360362 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.360769 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:50.360783 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:50.362139 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:50.364450 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.364455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.364458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.364689 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.364930 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.364933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.364935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.365000 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.365390 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:50.365392 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:50.365461 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 19:09:50.365565 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 19:09:50.365723 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260620 19:09:50.365974 24338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0: Neither blocks nor log segments found. Creating new log.
Jun 20 19:09:50 dist-test-slave-6wrf krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.154.189@KRBTEST.COM
I20260620 19:09:50.366459 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.356677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.1.154.188:41727)
0620 19:09:50.357197 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.357205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.357215 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:09:50.357248 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.357250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.357294 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.357377 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.357896 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.358636 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.359629 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.360081 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.360197 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.362254 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:50.362260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:50.362266 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:50.362297 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:50.364333 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.364784 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.364786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.364788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.364854 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.365079 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.365081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.365083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.365271 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:50.365359 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:50.366198 (+ 839us) server_negotiation.cc:299] Negotiation successful
0620 19:09:50.366312 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":112,"threads_started":1}
I20260620 19:09:50.367604 24338 log.cc:824] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:50.369289 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.353158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.154.188:41727 (local address 127.0.0.1:60300)
0620 19:09:50.353421 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:50.353436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:50.353506 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:50.354896 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:50.354899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:50.355116 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:50.355233 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.355241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.356594 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.356598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.357382 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:50.357391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.358383 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.358387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:50.358537 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.359340 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:50.359357 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:50.361053 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:50.367529 (+ 6476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.367532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.367534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.367780 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.368250 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:50.368253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:50.368255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:50.368357 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:50.368787 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:50.368790 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:50.368884 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 19:09:50.368975 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 19:09:50.369137 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":20}
I20260620 19:09:50.369704 24338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0: No bootstrap required, opened a new log
I20260620 19:09:50.370059 24490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5: Bootstrap starting.
I20260620 19:09:50.370405 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.353685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.1.154.188:41727)
0620 19:09:50.354705 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.354713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.354725 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:09:50.354756 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.354759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.354800 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.354881 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.355354 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.356468 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.357519 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.358157 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.358445 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.361203 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:50.361207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:50.361209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:50.361232 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:50.363471 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.367951 (+ 4480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.367957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.367959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.368072 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.368465 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.368468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.368470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.368665 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:50.369689 (+ 1024us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:50.369830 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 19:09:50.369951 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":104,"threads_started":1}
I20260620 19:09:50.373248 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.357075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.1.154.189:40481)
0620 19:09:50.357167 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.357170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.357181 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:09:50.357209 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.357212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.357245 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.358358 (+ 1113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.360893 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.361924 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.363421 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.363736 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.366703 (+ 2967us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.366831 (+ 128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:50.366834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:50.366837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:50.366855 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:50.370866 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.371373 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.371377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.371379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.371467 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.371841 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.371845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.371847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.372050 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:50.372221 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:50.372936 (+ 715us) server_negotiation.cc:299] Negotiation successful
0620 19:09:50.373048 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:09:50.373247 24490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:50.373723 24413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8: Bootstrap starting.
I20260620 19:09:50.374450 24490 log.cc:824] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:50.374918 24338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } }
I20260620 19:09:50.375198 24338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce5fa559e194da49f04e38f096dc5a0, State: Initialized, Role: FOLLOWER
I20260620 19:09:50.375838 24338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } }
I20260620 19:09:50.376140 24413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:50.376669 1642 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:09:50.376935 24413 log.cc:824] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:50.377208 24338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:50.377907 24312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5ce5fa559e194da49f04e38f096dc5a0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45276
I20260620 19:09:50.378144 24312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:09:50.378271 24312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:50.378580 24490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5: No bootstrap required, opened a new log
I20260620 19:09:50.379056 24413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8: No bootstrap required, opened a new log
I20260620 19:09:50.379621 24312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } }
I20260620 19:09:50.380010 24499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } } }
I20260620 19:09:50.380126 24499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:09:50.380589 24312 leader_election.cc:288] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [CANDIDATE]: Term 1 election: Requested vote from peers d95caded10694b32a38ea1468ab842b8 (127.1.154.189:40481), 20069f85bdae4fe68a230e792a365db5 (127.1.154.188:41727)
I20260620 19:09:50.383251 24413 raft_consensus.cc:348] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } }
I20260620 19:09:50.383539 24413 raft_consensus.cc:729] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d95caded10694b32a38ea1468ab842b8, State: Initialized, Role: FOLLOWER
I20260620 19:09:50.383886 24413 consensus_queue.cc:260] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } }
I20260620 19:09:50.385203 24413 sys_catalog.cc:564] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:50.385707 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ce5fa559e194da49f04e38f096dc5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d95caded10694b32a38ea1468ab842b8"
I20260620 19:09:50.385926 24386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:50.387257 24386 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce5fa559e194da49f04e38f096dc5a0 in term 1.
I20260620 19:09:50.387661 24288 leader_election.cc:302] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce5fa559e194da49f04e38f096dc5a0, d95caded10694b32a38ea1468ab842b8; no voters:
I20260620 19:09:50.388047 24499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:50.388351 24499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [term 1 LEADER]: Becoming Leader. State: Replica: 5ce5fa559e194da49f04e38f096dc5a0, State: Running, Role: LEADER
I20260620 19:09:50.388752 24499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } }
I20260620 19:09:50.384881 24270 master_runner.cc:186] Error getting master registration for 127.1.154.189:40481: OK, instance_id { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" instance_seqno: 1781982590094389 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:09:50.389007 24270 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:09:50.391010 24490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } }
I20260620 19:09:50.391253 24490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20069f85bdae4fe68a230e792a365db5, State: Initialized, Role: FOLLOWER
I20260620 19:09:50.391562 24490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } }
I20260620 19:09:50.392972 24490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:50.393247 24342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.382486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45282 (local address 127.1.154.190:43911)
0620 19:09:50.382598 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.382603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.383095 (+ 492us) server_negotiation.cc:407] Connection header received
0620 19:09:50.383377 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.383381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.383436 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.383488 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.384576 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.385197 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.386407 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.386716 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.386787 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.389093 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:50.389097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:50.389100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:50.389123 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:50.391053 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.391511 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.391514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.391516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.391601 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.391951 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.391954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.391956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.392173 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:50.392312 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:50.392972 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 19:09:50.393103 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:09:50.393677 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ce5fa559e194da49f04e38f096dc5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "20069f85bdae4fe68a230e792a365db5"
I20260620 19:09:50.393904 24464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:50.394578 24499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ce5fa559e194da49f04e38f096dc5a0. Latest consensus state: current_term: 1 leader_uuid: "5ce5fa559e194da49f04e38f096dc5a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } } }
I20260620 19:09:50.394687 24499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:50.395283 24532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:50.395417 24464 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce5fa559e194da49f04e38f096dc5a0 in term 1.
I20260620 19:09:50.395112 24345 master_runner.cc:186] Error getting master registration for 127.1.154.188:41727: OK, instance_id { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" instance_seqno: 1781982590314952 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:09:50.398952 24345 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:09:50.398900 24531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } } }
I20260620 19:09:50.399039 24531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:09:50.399768 24532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:50.403643 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.393983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.1.154.189:40481)
0620 19:09:50.394094 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.394100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.394467 (+ 367us) server_negotiation.cc:407] Connection header received
0620 19:09:50.394504 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.394506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.394548 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.394642 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.395672 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.396232 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.397449 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.397805 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.397899 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.399752 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:50.399759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:50.399778 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:50.399808 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:50.401511 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.401989 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.401992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.401994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.402104 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.402382 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.402385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.402387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.402617 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:50.402759 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:50.403392 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 19:09:50.403501 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 19:09:50.406651 24545 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:09:50.406733 24545 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 19:09:50.408215 24528 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:09:50.408281 24528 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:09:50.408447 24517 sys_catalog.cc:455] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } } }
I20260620 19:09:50.408560 24517 sys_catalog.cc:458] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:09:50.408758 24386 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8 [term 1 FOLLOWER]: Refusing update from remote peer 5ce5fa559e194da49f04e38f096dc5a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:50.408840 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:50.396645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.1.154.188:41727)
0620 19:09:50.397370 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.397374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.399375 (+ 2001us) server_negotiation.cc:407] Connection header received
0620 19:09:50.399549 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.399553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.399613 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.399665 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.400573 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.401402 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.402537 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:50.402882 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:50.402982 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:50.405004 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:50.405008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:50.405010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:50.405033 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:50.406754 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.407221 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.407225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.407226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.407311 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:50.407615 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:50.407618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:50.407621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:50.407822 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:50.407968 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:50.408562 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 19:09:50.408674 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636}
I20260620 19:09:50.410028 24464 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5 [term 1 FOLLOWER]: Refusing update from remote peer 5ce5fa559e194da49f04e38f096dc5a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:50.410365 24499 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:50.410781 24499 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:50.414309 24531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ce5fa559e194da49f04e38f096dc5a0. Latest consensus state: current_term: 1 leader_uuid: "5ce5fa559e194da49f04e38f096dc5a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } } }
I20260620 19:09:50.414426 24531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:09:50.416105 24517 sys_catalog.cc:455] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ce5fa559e194da49f04e38f096dc5a0. Latest consensus state: current_term: 1 leader_uuid: "5ce5fa559e194da49f04e38f096dc5a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } } }
I20260620 19:09:50.416268 24517 sys_catalog.cc:458] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:09:50.417740 24546 mvcc.cc:205] Tried to move back new op lower bound from 7299000690305400832 to 7299000690236268544. Current Snapshot: MvccSnapshot[applied={T|T < 7299000690305400832}]
I20260620 19:09:50.418926 24531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ce5fa559e194da49f04e38f096dc5a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } } }
I20260620 19:09:50.419057 24531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:09:50.419893 24499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ce5fa559e194da49f04e38f096dc5a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } } }
I20260620 19:09:50.420025 24499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:50.420125 24499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ce5fa559e194da49f04e38f096dc5a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } } }
I20260620 19:09:50.420202 24499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:50.420912 24517 sys_catalog.cc:455] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ce5fa559e194da49f04e38f096dc5a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce5fa559e194da49f04e38f096dc5a0" member_type: VOTER last_known_addr { host: "127.1.154.190" port: 43911 } } peers { permanent_uuid: "d95caded10694b32a38ea1468ab842b8" member_type: VOTER last_known_addr { host: "127.1.154.189" port: 40481 } } peers { permanent_uuid: "20069f85bdae4fe68a230e792a365db5" member_type: VOTER last_known_addr { host: "127.1.154.188" port: 41727 } } }
I20260620 19:09:50.421020 24517 sys_catalog.cc:458] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:09:50.421900 24550 mvcc.cc:205] Tried to move back new op lower bound from 7299000690305400832 to 7299000690236268544. Current Snapshot: MvccSnapshot[applied={T|T < 7299000690305400832}]
I20260620 19:09:50.422186 24549 mvcc.cc:205] Tried to move back new op lower bound from 7299000690305400832 to 7299000690236268544. Current Snapshot: MvccSnapshot[applied={T|T < 7299000690305400832}]
I20260620 19:09:50.424889 24532 catalog_manager.cc:1347] Generated new cluster ID: 5767b36af0f847b0ad5e3aaa71fcba7f
I20260620 19:09:50.424957 24532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:50.464843 24532 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:50.466825 24532 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:50.480015 24532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ce5fa559e194da49f04e38f096dc5a0: Generated new TSK 0
I20260620 19:09:50.480974 24532 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:09:50.518527 24419 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:50.515055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51354 (local address 127.1.154.189:40481)
0620 19:09:50.515193 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.515199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.516505 (+ 1306us) server_negotiation.cc:407] Connection header received
0620 19:09:50.517504 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.517509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.517573 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.517662 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.518376 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51354: BlockingRecv error: recv got EOF from 127.0.0.1:51354 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 19:09:50.520480 24495 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:50.515839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.1.154.188:41727)
0620 19:09:50.515977 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.515982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.519049 (+ 3067us) server_negotiation.cc:407] Connection header received
0620 19:09:50.519622 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.519626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.519682 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.519740 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.520348 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60318: BlockingRecv error: recv got EOF from 127.0.0.1:60318 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 19:09:50.521711 24342 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:50.516831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.1.154.190:43911)
0620 19:09:50.516969 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.516974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.519514 (+ 2540us) server_negotiation.cc:407] Connection header received
0620 19:09:50.520861 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.520865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.520935 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.520993 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.521531 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45286: BlockingRecv error: recv got EOF from 127.0.0.1:45286 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 19:09:50.526757 24419 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:50.524397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51358 (local address 127.1.154.189:40481)
0620 19:09:50.524518 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.524523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.525740 (+ 1217us) server_negotiation.cc:407] Connection header received
0620 19:09:50.525997 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.526000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.526084 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.526142 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.526655 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51358: BlockingRecv error: recv got EOF from 127.0.0.1:51358 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 19:09:50.532032 24419 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:50.530394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.1.154.189:40481)
0620 19:09:50.530501 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.530505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.531079 (+ 574us) server_negotiation.cc:407] Connection header received
0620 19:09:50.531275 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.531278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.531333 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.531389 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.531921 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51368: BlockingRecv error: recv got EOF from 127.0.0.1:51368 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 19:09:50.537837 24419 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:50.536149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.1.154.189:40481)
0620 19:09:50.536257 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.536261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.536821 (+ 560us) server_negotiation.cc:407] Connection header received
0620 19:09:50.537051 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.537054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.537108 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.537164 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.537725 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51384: BlockingRecv error: recv got EOF from 127.0.0.1:51384 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 19:09:50.545387 24419 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:50.543834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.1.154.189:40481)
0620 19:09:50.543961 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.543965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.544479 (+ 514us) server_negotiation.cc:407] Connection header received
0620 19:09:50.544658 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.544661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.544709 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.544759 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.545284 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51398: BlockingRecv error: recv got EOF from 127.0.0.1:51398 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 19:09:50.557446 24419 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:50.555745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.1.154.189:40481)
0620 19:09:50.555865 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.555869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.556452 (+ 583us) server_negotiation.cc:407] Connection header received
0620 19:09:50.556667 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.556670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.556725 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.556778 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.557334 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51400: BlockingRecv error: recv got EOF from 127.0.0.1:51400 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 19:09:50.578794 24419 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:50.576619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.1.154.189:40481)
0620 19:09:50.576765 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.576771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.577558 (+ 787us) server_negotiation.cc:407] Connection header received
0620 19:09:50.577845 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.577850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.577928 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.577997 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.578658 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51402: BlockingRecv error: recv got EOF from 127.0.0.1:51402 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 19:09:50.615414 24419 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:50.613454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.1.154.189:40481)
0620 19:09:50.613615 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.613621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.614263 (+ 642us) server_negotiation.cc:407] Connection header received
0620 19:09:50.614484 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.614489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.614547 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.614606 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.615284 (+ 678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51414: BlockingRecv error: recv got EOF from 127.0.0.1:51414 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 19:09:50.684080 24419 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:50.682623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.1.154.189:40481)
0620 19:09:50.682816 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.682821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.682984 (+ 163us) server_negotiation.cc:407] Connection header received
0620 19:09:50.683171 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.683174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.683231 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.683486 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.683968 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51426: BlockingRecv error: recv got EOF from 127.0.0.1:51426 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 19:09:50.816911 24419 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:50.815734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.1.154.189:40481)
0620 19:09:50.815890 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:50.815894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:50.816028 (+ 134us) server_negotiation.cc:407] Connection header received
0620 19:09:50.816173 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:50.816176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:50.816222 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:50.816307 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:50.816769 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51430: BlockingRecv error: recv got EOF from 127.0.0.1:51430 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 19:09:51.077153 24419 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:51.075781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.1.154.189:40481)
0620 19:09:51.075942 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:51.075946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:51.076059 (+ 113us) server_negotiation.cc:407] Connection header received
0620 19:09:51.076120 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:51.076122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:51.076171 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:51.076272 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:51.077036 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51444: BlockingRecv error: recv got EOF from 127.0.0.1:51444 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 19:09:51.408010 24545 catalog_manager.cc:1259] Loaded cluster ID: 5767b36af0f847b0ad5e3aaa71fcba7f
I20260620 19:09:51.408095 24545 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5: loading cluster ID for follower catalog manager: success
I20260620 19:09:51.409420 24528 catalog_manager.cc:1259] Loaded cluster ID: 5767b36af0f847b0ad5e3aaa71fcba7f
I20260620 19:09:51.409514 24528 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8: loading cluster ID for follower catalog manager: success
I20260620 19:09:51.409879 24545 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5: acquiring CA information for follower catalog manager: success
I20260620 19:09:51.410990 24545 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 20069f85bdae4fe68a230e792a365db5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:09:51.411109 24528 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8: acquiring CA information for follower catalog manager: success
I20260620 19:09:51.412045 24528 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d95caded10694b32a38ea1468ab842b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:09:51.597308 24623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:51.591642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.1.154.189:40481)
0620 19:09:51.591888 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:51.591892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:51.591908 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:09:51.591963 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:51.591965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:51.592086 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:51.592187 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:51.593015 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:51.593836 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:51.594915 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:51.595181 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:51.595402 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:51.595790 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:51.596492 (+ 702us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:51.596639 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:51.597092 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 19:09:51.597175 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260620 19:09:51.605043 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:51.601197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.1.154.188:41727)
0620 19:09:51.601415 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:51.601419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:51.601432 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:09:51.601515 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:51.601517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:51.601700 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:51.601806 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:51.602045 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:51.602591 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:51.603490 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:51.603721 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:51.603815 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:51.604174 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:51.604625 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:51.604731 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:51.604864 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 19:09:51.604913 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20260620 19:09:51.614163 24623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:51.608936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51476 (local address 127.1.154.189:40481)
0620 19:09:51.609070 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:51.609076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:51.609695 (+ 619us) server_negotiation.cc:407] Connection header received
0620 19:09:51.609980 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:51.609983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:51.610076 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:51.610138 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:51.610584 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:51.611389 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:51.612877 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:51.613217 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:51.613314 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:51.613653 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:51.613747 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:51.613818 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:51.613943 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 19:09:51.614000 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:09:51.614718 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:51.609070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45290 (local address 127.1.154.190:43911)
0620 19:09:51.609373 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:51.609380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:51.609543 (+ 163us) server_negotiation.cc:407] Connection header received
0620 19:09:51.609904 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:51.609909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:51.610092 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:51.610208 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:51.610502 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:51.611367 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:51.612374 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:51.612667 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:51.612767 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:51.613109 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:51.613823 (+ 714us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:51.613939 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:51.614485 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 19:09:51.614546 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260620 19:09:51.615875 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:51.609394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.1.154.188:41727)
0620 19:09:51.609517 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:51.609523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:51.609734 (+ 211us) server_negotiation.cc:407] Connection header received
0620 19:09:51.609946 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:51.609949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:51.609995 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:51.610079 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:51.610566 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:51.611139 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:51.612684 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:51.613025 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:51.613123 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:51.615348 (+ 2225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:51.615435 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:51.615520 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:51.615698 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 19:09:51.615759 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:09:51.618932 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 24270
I20260620 19:09:51.625293 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 24345
I20260620 19:09:51.631757 1642 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8BnTdh/build/debug/bin/kudu with pid 24420
2026-06-20T19:09:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3174 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6471 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (298222 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST